Masaryk UniversityFaculty of Informatics

Cloud Integration ofContinuous IntegrationDeclarative Pipelines

Master’s Thesis

Denis Richtárik

Brno, Spring 2018

Replace this page with a copy of the official signed thesis assignment and acopy of the Statement of an Author.

Declaration

Hereby I declare that this paper is my original authorial work, whichI have worked out on my own. All sources, references, and literatureused or excerpted during elaboration of this work are properly citedand listed in complete reference to the due source.

Denis Richtárik

Advisor: Bruno Rossi Ph.D.

i

Acknowledgements

I would like to thank my tutor Bruno Rossi, Ph.D. for his valuable notes,help and patience. I feel honored to work under his guidance. I wouldalso like to thank my colleague Oliver Gondža, who always pointedme the right direction and never refused to answer my questions.

iii

Abstract

This thesis provides an implementation of a Declarative Pipeline inOpenStack Cloud Plugin for Jenkins. It is the next step in Configurationas Code methodology in this plugin and introduces another level ofautomation in its infrastructure. The theory behind concerned topicsand principles is explained in a scope relevant to the problem it solves.

iv

Keywords

Cloud computing, Continuous Integration, Jenkins Pipeline, Declara-tive Pipeline, Configuration as Code, Red Hat, OpenStack

v

Contents

1 Introduction 11.1 The goals of the thesis . . . . . . . . . . . . . . . . . . . . . 11.2 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . 2

2 Continuous Integration and Declarative Pipelines 32.1 Automation processes in software development . . . . . . . 3

2.1.1 Automation in software building . . . . . . . . . 42.2 Configuration as Code . . . . . . . . . . . . . . . . . . . . 4

2.2.1 Principles of Configuration as Code . . . . . . . 52.2.2 Goals of Infrastructure as code . . . . . . . . . . 6

2.3 Continuous Integration . . . . . . . . . . . . . . . . . . . . 72.3.1 Continuous integration basic tools . . . . . . . . 82.3.2 Benefits of Continuous Integration . . . . . . . . 92.3.3 Continuous Delivery . . . . . . . . . . . . . . . . 102.3.4 Establishment . . . . . . . . . . . . . . . . . . . . 11

2.4 Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.4.1 A deployment pipeline . . . . . . . . . . . . . . . 12

2.5 Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.5.1 From Hudson to Jenkins . . . . . . . . . . . . . . 142.5.2 Jenkins plugins . . . . . . . . . . . . . . . . . . . 15

2.6 Jenkins Pipeline . . . . . . . . . . . . . . . . . . . . . . . . 162.6.1 Why Jenkins Pipeline . . . . . . . . . . . . . . . . 172.6.2 Declarative pipeline . . . . . . . . . . . . . . . . 182.6.3 Agent . . . . . . . . . . . . . . . . . . . . . . . . . 192.6.4 Comparison to Scripted Pipeline . . . . . . . . . 202.6.5 Other Declarative Pipeline options . . . . . . . . 212.6.6 Nomenclature . . . . . . . . . . . . . . . . . . . . 22

2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3 Cloud Computing 233.1 Software as a Service . . . . . . . . . . . . . . . . . . . . . 23

3.1.1 Cloud as a service . . . . . . . . . . . . . . . . . 243.2 Benefits of Cloud Platforms . . . . . . . . . . . . . . . . . . 243.3 Cloud elasticity . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.1 Amazon EC2 . . . . . . . . . . . . . . . . . . . . 25

vii

3.4 OpenStack . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4 Problem statement 294.1 Problem description . . . . . . . . . . . . . . . . . . . . . . 294.2 Solution design . . . . . . . . . . . . . . . . . . . . . . . . 29

4.2.1 Learn the theory . . . . . . . . . . . . . . . . . . 294.2.2 Start implementing . . . . . . . . . . . . . . . . . 30

5 Development 335.1 Initial preparation . . . . . . . . . . . . . . . . . . . . . . . 335.2 Maven configuration . . . . . . . . . . . . . . . . . . . . . 33

5.2.1 Adding dependencies . . . . . . . . . . . . . . . 345.3 Test-driven development . . . . . . . . . . . . . . . . . . . 355.4 Implementing an OpenStack agent . . . . . . . . . . . . . . 35

5.4.1 Agent options . . . . . . . . . . . . . . . . . . . . 365.5 A Pipeline Step . . . . . . . . . . . . . . . . . . . . . . . . 37

5.5.1 Changes in the original code . . . . . . . . . . . 385.5.2 Step Execution . . . . . . . . . . . . . . . . . . . 38

5.6 Groovy script . . . . . . . . . . . . . . . . . . . . . . . . . 395.7 Class diagram . . . . . . . . . . . . . . . . . . . . . . . . . 405.8 Encountered issues . . . . . . . . . . . . . . . . . . . . . . 41

5.8.1 Names of plugins . . . . . . . . . . . . . . . . . . 415.8.2 Poor documentation of many plugins . . . . . . 415.8.3 Declarative Pipeline DSL . . . . . . . . . . . . . 415.8.4 Wrong version of Jenkins . . . . . . . . . . . . . 42

5.9 Production testing . . . . . . . . . . . . . . . . . . . . . . 425.9.1 Environment for plugin development testing . . 425.9.2 Standalone Jenkins deployment . . . . . . . . . . 435.9.3 Setting up an OpenStack connection . . . . . . . 435.9.4 Setting up a job . . . . . . . . . . . . . . . . . . . 445.9.5 Running a build . . . . . . . . . . . . . . . . . . . 44

5.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6 Conclusion 49

Bibliography 51

viii

A Final version of test Jenkinsfile 55

B OpenStack Declarative Agent 57

C Class diagram 59

ix

List of Figures

2.1 CI vs CD comparison. Source: [13] 112.2 A deployment pipeline. Source: [12] 132.3 Uploading a plugin directly into Jenkins 152.4 Jenkins Pipeline technologies. Source: Alex Soto, Red

Hat 173.1 Traditional vs elastic cloud computing. Source: [24] 263.2 What is OpenStack. Source: [26] 274.1 Components diagram 304.2 Sequence diagram 315.1 Check if a Declarative Pipeline functionality 355.2 Mock and check of a new machine creation 365.3 OpenStackNodeStepExecution 395.4 Passing of OpenStack arguments from Jenkinsfile to the

Step 405.5 Directly installing the plugin into Jenkins 435.6 Configuration used for a real build 445.7 Successful build in Jenkins 455.8 Temporary server available in Jenkins 465.9 Agent created in Red Hat’s instance of OpenStack 46

xi

1 Introduction

This thesis is about automation in software development, ContinuousIntegration and Configuration as Code in particular. My bachelor’sthesis was my introduction to software development automation andstarted my professional career in this field. Automation, in general,has been evolving for a few decades now and it was only logical that itwould become an important part of software engineering. Similar as inother fields, organizations are trying to save money and developmentteams are trying to save time and valuable resources. Because automa-tion is a difficult process, it cannot happen suddenly. At least on insoftware development. Jenkins has been around for nearly 15 yearsand still is one of the most used automation tools for software. Andwith its monthly releases, more and more features are introduced.

And it is not only about the tools and their capabilities. It is aboutevery product and how each of this product embraces automation.When a team starts with automation too late, it has much harder timeto automate their existing processes.

Even though automation is an ongoing and slow process, thisthesis helps us to go another step further. It helps to utilize new andmodern concepts and implement them in a way that it can be linkedwith a product that is widely used and has potential to help to betterautomate many enterprise products in the world.

Configuration as Code, to some degree, reinvented Jenkins. It camewith default implementations for Docker and Kubernetes. This thesisbrings these possibilities also to OpenStack.

1.1 The goals of the thesis

The main goal of this thesis is to provide a Declarative Pipeline featurefor OpenStack Cloud Plugin of Jenkins. The implementation includesa second main goal - to implement a solution where a user can definehis own agent parameters for an OpenStack machine without havingto ask an administrator.

The third goal, besides the implementation part, was to defineand explain the theory behind the problem so that a reader, withoutdetailed insight to the problem, would be able to understand the

1

1. Introduction

topic and problem it solves in general. Another goal was to comparedeclarative and imperative approaches to pipeline configurations andexplain what declarative approach needs to be implemented.

1.2 Structure of the thesis

In a following chapter 2, the theory behind software developmentautomation and Continuous Integration will be explained. In chapter3 the theory behind Cloud computing, its elasticity and OpenStackwill be provided.

Chapter 4 will explain the exact implementation problems thethesis will try to solve and the methodology used to solve them. Itwill contain diagrams as a helpful tool to visualize the problem andsequence of the solution pattern.

In chapter 5 the process of development are explained with samplesof code to better understand the problem and code structure.

In the last chapter 6, the final conclusion will recapitulate the thesisand reflect its result.

2

2 Continuous Integration and Declarative Pipe-lines

This chapter explains principles and theory related to the goal of thisthesis. The scope and amount of information are adapted so that itdoes not go too deep into unnecessary detail but provides enoughinformation to understand the context of the problem this thesis solves.

The chapter begins with describing automation in software de-velopment and importance of using a code for configurations and itsprinciples. Next, it describes Continuous Integration methodology insoftware development and its important benefits.

The chapter will also describe what a pipeline is, specifically adeployment pipeline and types of pipelines in Jenkins. It briefly intro-duces Jenkins, its history and key parts of this thesis.

2.1 Automation processes in software development

Bran Selic in his article The Pragmatics of Model-Driven Development[1] says that automation is by far the most effective technologicalmeans for boosting productivity and reliability.

Over time, inevitably, software engineering has become a complexscience. Modern software is usually significantly large with manypeople, tools, principles or methods involved. Some processes becamerepetitive, some less secure and some obsolete. We developed frame-works which ease our work and we introduced automation to softwareengineering as a natural step forward in reinventing it.

Automation does not only mean increasing productivity but alsoincreasing the quality of software. But to achieve such a goal is nevereasy. To automate means to collaborate with other team members,possibly other teams. It means to introduce new tools into the workflow.It also means that somebody is going to need to learn new techniquesand technologies.

Paul Grünbacher and Yves Ledru [2] listed these software engi-neering areas that are being automated: issue tracking, specification,design, implementation, testing, verification, maintenance, configura-tion management, delivery, deployment and more.

3

2. Continuous Integration and Declarative Pipelines

2.1.1 Automation in software building

Because this thesis is aimed at automating build processes of softwaredevelopment, it is going to explain theories and terms involved aroundintegrating and delivering a project from its source code to a usableproduct.

Alfonso Fuggetta and Elisabetta Di Nitto in their article SoftwareProcess [3] explain a few examples of automation of software building.When it comes to building a software product, it is possible to automateevery process that is related to building, deploying and operating.These are the examples they provide:

∙ Anyone with an access to the repository of project’s versioncontrol system is able to download all the code sources, exter-nal libraries and components and deploy the final product byrunning a Maven script.

∙ There are tools and frameworks to use for automation. Jenkinsas a framework, for example, utilizes a version managementsystem. When a change is pushed to a linked repository, it isdetected by the framework and a preconfigured set of actionsare triggered. These actions are processed in a defined environ-ment and close the developing, integrating, quality checkingand building cycle. This processes then gives collected, struc-tured and important information about project’s developmentstability and health. This framework can, if requested, providea deployable product.

∙ With the arrival of virtualization, cloud and hybrid cloud, au-tomation has a big impact on the operation phase. AlfonsoFuggetta and Elisabetta Di Nitto [3] mention mainly applicationmanagement and its related software. An example: for applica-tion management, a tool called Puppet is being used and Nagiosfor software’s state monitoring.

2.2 Configuration as Code

Configuration as code originates in term Infrastructure as code. AdamBertram says in his article [4] that Infrastructure as Code, or IaC, may

4

2. Continuous Integration and Declarative Pipelines

sound misleading. He explains that cloud providers have physicalinfrastructures. For example, Azure by Microsoft or Amazon Web Ser-vices. These infrastructures can be rented from these providers or anorganization may have its own. Infrastructure as Code means it is au-tomation and management, not replacing physical resources entirely.Bertram says that even though Configuration as Code is more techni-cally accurate, Infrastructure as Code became the standard name.

To provide a more simple explanation, Jafari Sitakange [5] explainsConfiguration as Code as a programmable infrastructure that allowsus to write a code in any descriptive programming language. Thiscode is then used for configuration management and automation ofsaid infrastructure provisioning and its deployment.

Facebook, in their research [6], proved that writing code to gen-erate configurations reduces the burden of manually keeping trackof changes, thanks to a code reuse. One of the reasons related toautomation is that compiled configurations are generated from theconfiguration source code, and it changes 60% more frequently thanwith the classic approach, because the change of one source code filemay generate multiple new compiled configurations.

There are several automation tools for configuration management.For example, Ansible, Chef, or Puppet. They allow configuration ofinfrastructures with specialized programming languages sometimescalled domain-specific languages. DevOps teams can then use theirtools along with a Version Control System or principles like Continu-ous Integration, Continuous Delivery, or Continuous Deployment.

2.2.1 Principles of Configuration as Code

Kief Morris in his book Infrastructure as Code [7] defined these prin-ciples:

∙ Reproducibility - Any part of an infrastructure should be ableto be rebuilt at any time and without problems. The environmentparameters of a previous build are stored and available.

∙ Consistency - If there are two elements of infrastructure thatare set up with two identical services, then the servers should benearly identical. They should be configured in an identical man-

5

2. Continuous Integration and Declarative Pipelines

ner and the only difference should be in necessary parameters,such as an IP address.

∙ Repeatability - Relies on the first principle of reproducibilityand means that any action previously run should be able to berepeated.

∙ Disposability - Provides the option to destroy any element ofthe infrastructure without hassle at any time.

∙ Service continuity - Any service that is part of concrete infra-structure must be available non-stop and should not be damagedwhen individual infrastructure elements disappear.

∙ Self-testing systems - Morris declares automated testing as oneof the most important practices in automation and infrastructureoperations and that teams should adopt them from the verybeginning of development.

∙ Self-documenting systems - It is a persistent problem in soft-ware engineering to maintain relevant, high-quality documen-tation. With automation, these goals are easier to achieve andat least provide the ability to focus on new content.

∙ Small changes - It is easier to maintain, test, and fix smallchanges. It gives the code better structure when a big pieceof work is divided into incremental changes.

∙ Version all the things - Versioning the configuration of theinfrastructure is as important as versioning any code or software.

2.2.2 Goals of Infrastructure as code

Kief Morris [7] also defined these goals of Infrastructure as code:

∙ IT infrastructure is a principle which provides change and notthe opposite.

∙ The software engineering team involved in a particular projectshouldn’t be wasting time on redundant and repetitive tasks.

6

2. Continuous Integration and Declarative Pipelines

∙ Customers or users should be able to manage their resourceswithout having software engineers doing it for them.

∙ Recovering from a failure should not be as catastrophic so thatit is absolutely necessary to avoid them.

∙ Changes to the configuration are incremental, easy and wel-comed.

∙ It is important to change the configuration continuously and insmaller chunks.

∙ Providing new features or fixes should be done via implement-ing, testing and measuring. These processes, not meetings ordocuments, should provide their reliability.

2.3 Continuous Integration

Continuous Integration is a software development practice wheremembers of a team integrate their work frequently - usually, each per-son integrates at least daily - leading to multiple integrations per day[8]. Then, by using continuous integration tools, these integrations areautomatically verified and tested. This approach allows developmentteams to detect bugs and issues as soon as possible. A well-designedcontinuous integration infrastructure helps to boost teams’ produc-tivity and ease the processes from committing the feature change todeploying the whole product.

It has been proven by many projects and software engineeringteams that continuous integration allows more agile and rapid soft-ware development. This kind of infrastructure usually provides specialgraphical user interfaces to provide data and operations, which arefocused and understandable both by a developer and a customer/user.John Ferguson Smart, in the book Jenkins: The Definitive Guide, ex-plains that continuous integration helps teams deliver more real busi-ness value to the end user and that every professional developmentteam, no matter how small, should be practicing it [9].

7

2. Continuous Integration and Declarative Pipelines

2.3.1 Continuous integration basic tools

There are not many requirements for tools to do continuous integration.The two basic tools described in this section are source control systemand build tool.

Authors of a book Continuous Integration [10] present this as atypical Continuous Integration scenario:

1. The scenario starts with a developer committing a code to theversion control repository. The changes are then detected by acontinuous integration server, that regularly queries the reposi-tory.

2. The server pulls the changes to its local repository - a copy ofthe original. Afterward, a build script is triggered to execute thebuild, tests and verification.

3. The continuous integration server then sends e-mails to a mail-ing list, server administrator or the contributor about the buildresult. The result is also stored and provided for statistics.

4. The continuous integration server goes back to detecting changesin the version control repository.

This scenario includes those two tools described next.

Source control system

In software development, the basic tool is source or version controlsystem, which helps to manage the code and puts it into a repository.It also helps to separate streams of development flows. Usually, thereis one main steam called master.

A contributor or a developer of the software can download thesource code, make any changes they want, commit their changes andthen push the changes into the source control system where othercontributors can then download (pull) the changes other contributorsdid since their latest update of the stream.

By committing changes, the contributor says what his change doesand encapsulates the change into a record. In the source control system,developers can see these records list of a particular repository. Before

8

2. Continuous Integration and Declarative Pipelines

committing, the contributor also needs to build the changed softwareto check if the code compiles and also has to run tests. Having well-written tests with a good coverage is an important part of continuousintegration. Otherwise many bugs and issues would come up withnot properly tested commits.

Build tool

Another tool that is widely used in continuous integration of softwareis a build tool which can detect changes in a code form linked versioncontrol repository and run a build with a test. This is important be-cause such automated build usually provides much more complexprocesses which could be run on different environments compared toa developer who, before committing his or her core runs just the basicset of tests. This complex flow of instructions often results in a pro-duction build that might require such tests on various environmentsto guarantee functionality for customers or end users.

The automation tool then keeps a track of build success rate andsends notifications about problems in a build in case of bad integration.Such a tool is Jenkins and is described in detail in section 2.5.

2.3.2 Benefits of Continuous Integration

A production build, which is a final build intended for end user orcustomer, is a complex set of processes and because of that, it is hardto predict whether the build will be successful. With continuous inte-gration of the code changes and frequent test production builds, thedevelopment team can predict the delivery time of the product whichlowers the possibility of a delayed release. Build success rate and buildissues frequency are recorded what provides software health. Thesedata can be used for statistics and future improvements.

Because merging of everyone’s work on the same project into onestream with a frequent build and dedicated tests runs, the discoveryof bugs is faster and thus can be fixed before it is too late. This isimportant when there is a bug that has a major impact on the codefunctionality. Also Facebook, in their article Holistic ConfigurationManagement at Facebook [6] says that they release software early andfrequently. They say that they use continuous integration to get fast

9

2. Continuous Integration and Declarative Pipelines

feedback and integrate rapidly. Because the time between two releasesis short, it makes troubleshooting easier for them.

Another benefit is reducing repetitive manual processes. Depend-ing on the level of the build automation, developers usually have todo repetitive tasks every production build. These processes can bevery error-prone on the developer’s side. Automating these processessaves a lot of time and effort which could be used somewhere else.

One of the less mentioned benefits is more collaboration inside ateam or between teams or subteams. As it was proved many times, col-laboration and communication bring many benefits into software de-velopment and continuous integration methodology, without propercommunication, is impossible.

2.3.3 Continuous Delivery

Continuous Delivery is a software engineering approach in whichteams keep producing valuable software in short cycles and ensurethat the software can be reliably released at any time [11].

Continuous Delivery is becoming more and more popular. Byadopting this methodology, software engineering teams and compa-nies in general can bring final product or service to a customer. Butthis is not limited only to big releases but continuous delivery canalso mean very fast and small improvements. In case of major issues,deploying a product with a patch or new feature quickly, mean a bigadvantage in competition.

But implementing Continuous Delivery is a difficult process whetherit is an existing product or a new one. Also, the more complex the prod-uct is, the more difficult it is to automatize the continuous deliveryprocess.

The biggest benefit of Continuous Delivery is a rapid feedbackfrom the customer. This feedback provides faster issue reports andthus their fixes compared to a classic delivery process, for example, twotimes a year. The concept of continuous delivery is closely related tothe concept of continuous integration [12]. The differences are shownin Figure 2.1.

10

2. Continuous Integration and Declarative Pipelines

Figure 2.1: CI vs CD comparison. Source: [13]

2.3.4 Establishment

The term Continuous Integration was introduced by Grady Booch in1999 in his book Object-Oriented Analysis and Design with Applica-tions [14]. Afterward, it became a part of the Extreme Programmingdevelopment process, as one of its original twelve practices [8]. InExtreme Programming, the aim was to automate tests as part of test-driven development. Automation of the whole build came later, stepby step.

The concept of Continuous Delivery was established during theyear 2010 by Jez Humble and David Farley [12], who wrote a collabo-rative book about it. It was proposed as an extension for ContinuousIntegration and gained wide popularity. The concept is relatively new.

2.4 Pipeline

The pipeline breaks down the software delivery process into stages.These stages are focused on verification of the quality of the code tovalidate the new functionality, its compatibility with the code and pre-vent issues that could come up and affect the end user. The pipelineprovides regular feedback and stores a change history of its config-

11

2. Continuous Integration and Declarative Pipelines

uration. These data can be available to anyone in the developmentteam.

There is no such thing as The Standard Pipeline, but a typicalCD pipeline will include the following stages: build automation andcontinuous integration; test automation; and deployment automation[15].

∙ Build automation and Continuous Integration: The pipelinebuilds the code and its binaries to create a deliverable product.This product is then acted upon in next stages. This processis triggered manually or automatically when a change in thecode repository is detected. Typically when a new feature isimplemented. This stage also contains unit testing. The buildstage is the most major health indicator of the project.

∙ Test Automation: In this stage, the product is being tested toprovide a high degree of quality assurance. Whether it is func-tionality, security, performance or compliance, all need to beverified. Test automation can still include some manual activitiesas it is not easy, or sometimes even possible, to automate sometests. Some tests may be manual because there is not enoughtime or complexity is too high such as user interface testing.

∙ Deployment Automation: Phillips writes about the importanceof rollout time and that it is the most critical moment for de-ployment automation. This stage has lower failure rates thanthe previous stages because the product has already been builtand tested. The product now can be deployed to a staging repos-itory, verified by the release team and moved to a productionenvironment

2.4.1 A deployment pipeline

A deployment pipeline is, in essence, an automated implementation ofyour application’s build, deploy, test, and release process [12]. Everypipeline with its stages is different. There are numerous implemen-tations and recommendations but the principles mentioned in thisthesis are usually the same and should be respected. Jez Humble and

12

2. Continuous Integration and Declarative Pipelines

Figure 2.2: A deployment pipeline. Source: [12]

David Farley provide an example of a typical deployment pipeline. Itcan be seen in Figure 2.2.

A deployment pipeline is a part of the continuous deploymentprinciple. Continuous deployment enables a project to deliver working,tested, and deployable software at any point in time whenever it isrequested. This means a key purpose of a Continuous Integrationsystem is to generate the bundled software artifacts with the latestcode changes and make it available to a testing environment [10].

2.5 Jenkins

Jenkins, originally called Hudson, is an open source Continuous Inte-gration tool written in Java. Boasting a dominant market share, Jenkinsis used by teams of all sizes, for projects in a wide variety of languagesand technologies.

Jenkins is easy to use. The graphical user interface is simple, intu-itive, and visually appealing, and Jenkins as a whole has a very lowlearning curve [9].

It is a platform used for continuous integration and continuousdelivery by many software companies around the world.

Because Jenkins is an enormous community project, the develop-ment cycles are frequent, bugs get fixed faster, and there is alwayssomeone willing to help with user or development problems. If some-one needs help with Jenkins related questions, he or she can contactthe community via the IRC channel on Freenode or use the appro-priate mailing list. Both of these communication channels are veryactive.

13

2. Continuous Integration and Declarative Pipelines

Frequent development cycles produce new releases weekly, but forenterprises or users with requirements for stability and security, thereare long-term support versions with releases about once a month.

2.5.1 From Hudson to Jenkins

Jenkins was started in the late year 2004 by a Sun Microsystems devel-oper and employee Koshuke Kawaguchi. Back then, the project wascalled Hudson. In the upcoming years, Hudson became increasinglypopular inside Sun Microsystems and many teams were integrating itin their processes. By 2010, Hudson had become the leading Continu-ous Integration solution with a market share of over 70% [9].

During 2009 and finally in January 2010, Oracle acquired Sun Mi-crosystems. This lead to many problems inside original teams. Themain problem was that Oracle trademarked the name Hudson in late2010, along with other problems such as management disagreements.Oracle also wanted to change the development model to more slowerand more controlled which lead to a lot of frustration inside the orig-inal Hudson team lead by Kohsuke. The team wanted to continueusing their proven development process which worked well so far. Itwas open, flexible and community-focused with rather often releases.

In January 2011, there was a vote in Hudson community to renamethe project to Jenkins. It was moved to a GitHub repository as a newproject and Kohsuke Kawaguchi with most of the core developersalong with community members moved their focus and developmentto this project.

Oracle claimed that Jenkins was only a fork of Hudson and con-tinued their work. But because it lacked the community and usersslowly started switching to Jenkins. In mid-2011, polls showed thatabout 75% of Hudson users had switched to Jenkins, while 13% werestill using Hudson, and another 12% were using both Hudson andJenkins or in the process of migrating to Jenkins [9].

Later, Hudson became part of Eclipse foundation. Oracle could notcompete with Jenkins so they let go the source code and trademark.In January 2013, it had its first major release 3.0 as part official part ofEclipse Foundation.

14

2. Continuous Integration and Declarative Pipelines

Figure 2.3: Uploading a plugin directly into Jenkins

2.5.2 Jenkins plugins

The basic characteristic of Jenkins is its dependency on plugins. Nearlyeverything in Jenkins is done by plugins. There are close to 1500plugins available. If it is git functionality, Maven integration, sendinge-mails or Pipeline, it is all done by extending the Jenkins core libraries.

When Jenkins is installed, the user is asked if he or she wants toinstall the default set of plugins or choose his own to install a specificflavor of this tool. Once done and installed, in Jenkins configuration isplugins manager for installing, removing or deleting plugins.

There is an advanced option to upload a specific plugin in hpipackaging. This is a feature widely used when a user needs a specificversion of a plugin. In some cases, Jenkins may automatically down-load a plugin as a transitive dependency with a version that is notdesired by the user.

Another use case for uploading a plugin directly is to test a newversion of a plugin in a clean environment. By building the projectwith Maven Jenkins Plugin, the plugin is packaged into a hpi package,which can be uploaded and tested on a locally installed instance ofJenkins. By the end of the development phase of this thesis, this func-tionality is going to be used. The option to upload a plugin directly isshown in Figure 2.3.

Jenkins types of jobs

A job in Jenkins can be also called an item. It is a starting point whensetting up a job to run user-defined builds or steps. A Jenkins instancewith a default set of plugins installed provides these 6 kinds of items:

∙ Freestyle project - A major feature of Jenkins. It includes build-ing a project using any type of build system, integration with

15

2. Continuous Integration and Declarative Pipelines

version control system, sending emails or triggering multiplebuild scripts. It is the most general job and is not limited to be asoftware build.

∙ Pipeline - Orchestrates long-running activities that can spanmultiple build agents. Suitable for building pipelines (formerlyknown as workflows) and/or organizing complex activities thatdo not easily fit in free-style job type.

∙ Multi-configuration project - Suitable for projects that needa large number of different configurations, such as testing onmultiple environments, platform-specific builds, etc.

∙ Folder - Creates a container that stores nested items in it. Usefulfor grouping things together. Unlike view, which is just a filter,a folder creates a separate namespace, so you can have multiplethings of the same name as long as they are in different folders.

∙ GitHub Organization - Scans a GitHub organization (or useraccount) for all repositories matching some defined markers.

∙ Multibranch Pipeline - Scans an SCM repository and creates aPipeline for each detected branch.

2.6 Jenkins Pipeline

At the end of April 2016, Mr. Kohsuke Kawaguchi introduced Jenkinsversion 2 with a feature called Pipeline as Code. Up until then, Jenkinshad had Pipeline possibilities but were limited to graphical user inter-face configuration with each execution step configured as a JenkinsJob. These Jenkins Jobs were then tied by on option to trigger anotherbuild based on a condition or build result of a currently executed build.With Jenkins 2.0, Pipeline Domain Specific Language based on Groovylanguage was introduced. With this release, the Pipeline plugin alsobecame a part of the default set of plugins for Jenkins.

Introduction of Jenkins Pipeline meant that developers could usea unified language that could be used to define complex pipeline jobsinside one configuration file called Jenkinsfile. Jenkinsfile can be eitherconfigured inside the graphical user interface of Jenkins or managed

16

2. Continuous Integration and Declarative Pipelines

Figure 2.4: Jenkins Pipeline technologies. Source: Alex Soto, Red Hat

through a Source configuration management such as git. GitHub sup-port was introduced with branches management, detecting changeswhich could be acted upon. This new features allowed automatic setupof any Pipeline defined by a Jenkinsfile.

With a proper documentation explaining Pipeline syntax, it is noweasy to create, modify or remove pipelines by any team member withaccess rights.

The release also supported graphical interface for a Pipeline build,which displayed execution of each job in a nice and well readablemanner.

Some of the technological possibilities of Jenkins Pipeline areshown in Figure 2.4.

2.6.1 Why Jenkins Pipeline

∙ It is a great and easy-to-use tool for development teams to in-tegrate, test and deploy the project’s code. They can set up asource code repository and release a built product as manytimes as they want. All via Jenkins.

∙ Configuration as Code or Pipeline as Code in Jenkins is managedthrough Jenkinsfiles which makes management of Pipelineseasy, fast and unified. These configuration files can be stored in

17

2. Continuous Integration and Declarative Pipelines

SCM. In older versions of Jenkins, before the Pipeline, a userhad to define the pipeline workflow jobs in the graphical userinterface. Jenkinsfile provides better maintenance, reliability,and reproduction.

∙ There is an option to define Multi-branch pipelines to differ-entiate multiple configurations. For example, there could bedifferent configurations for patch builds, rpm builds, produc-tion builds or lightweight test builds.

∙ Even team members who are not programmers can use JenkinsPipeline to trigger their own builds by checking out the Jenk-insfile from SCM to their Jenkins instance or use a providedone. With user input parameters with descriptions, it is easyto understand what the Pipeline requires and then trigger thebuild.

2.6.2 Declarative pipeline

In March 2017 a team around Andrew Bayer, a CloudBees developer,introduced a Declarative Pipeline Plugin which simplified the DomainSpecific Language of Jenkins Pipeline. It was released with Jenkinsversion 2.5.

His idea was to introduce a simpler syntax via declarative sectionblocks all wrapped in a pipeline { ... } block. The section blocks vary.There can be a stages block or agent block. Stages contain one ormore stages blocks. A stage has a name and behaves similarly to abuild step in a regular Jenkins job. Usually, and by a recommendation,stages include at least one stage. Each stage then has steps which areexecuted.

The basic statements and expressions which are valid in Declar-ative Pipeline follow the same rules as Groovy’s syntax with the fol-lowing exceptions:

∙ The top-level of the Pipeline must be a block, specifically: pipeline

∙ No semicolons as statement separators. Each statement has tobe on its own line.

18

2. Continuous Integration and Declarative Pipelines

∙ Blocks must only consist of Sections, Directives, Steps, or assign-ment statements.

∙ A property reference statement is treated as a no-argumentmethod invocation. So for example, input is treated as input()[16]

2.6.3 Agent

An agent defines where the Pipeline is run. It can be defined as onedefault agent for all of the stages with the possibility to define a dif-ferent agent for a particular stage. If the default agent is not defined(agent none), then each stage must define its own.

Currently, there are these 6 agent parameters available:

∙ any - run on any available agent (node)

∙ none - define an agent in each stage

∙ label - run on a specific agent with a provided label name

∙ node - behaves similar to agent label but allows options to it

∙ docker - run on any node that contains a specified Docker con-tainer

∙ dockerfile - execute the Pipeline on a container defined from aDockerfile

Example of agent any:

pipeline {agent anystages {

stage(’Example ’) {steps {

echo ’Hello␣World’}

}}

}

19

2. Continuous Integration and Declarative Pipelines

Example of agent docker and using a different agent for each stage:

pipeline {agent nonestages {

stage(’Example␣Build’) {agent { docker ’maven:3 -alpine ’ }steps {

echo ’Hello ,␣Maven’sh ’mvn␣--version ’

}}stage(’Example␣Test’) {

agent { docker ’openjdk:8 -jre’ }steps {

echo ’Hello ,␣JDK’sh ’java␣-version ’

}}

}}

2.6.4 Comparison to Scripted Pipeline

Where they differ is in syntax and flexibility. Declarative limits whatis available to the user with a more strict and predefined structure,making it an ideal choice for simpler continuous delivery pipelines.Scripted provides very few limits, that the only limits on structure andsyntax tend to be defined by Groovy itself, rather than any Pipeline-specific systems, making it an ideal choice for power-users and thosewith more complex requirements. As the name implies, DeclarativePipeline encourages a declarative programming model. Whereas ScriptedPipelines follow a more imperative programming model.

As mentioned before, there is another way to define a JenkinsPipeline. It is a different DSL called Scripted Pipeline syntax. It wasreleased with the first Pipeline in Jenkins 2.0. With Jenkins 2.5 thesetwo approaches were separated to Scripted Pipeline syntax and De-

20

2. Continuous Integration and Declarative Pipelines

clarative Pipeline syntax. The former one is still supported but not aspromoted and recommended as the Declarative approach.

Scripted pipeline syntax offers more flexibility. Declarative is lim-ited due to a strict and predefined structure. While Declarative Pipelineuses declarative programming model, Scripted Pipeline is more im-perative.

Declarative pipeline can still use the syntax of Scripted pipelineinside of special script blocks as shown on the following snippet:

Jenkinsfile (Declarative Pipeline)

pipeline {agent anystages {stage(’Example ’) {steps {echo ’Hello␣World’

script {def browsers = [’chrome ’, ’firefox ’]for(int i = 0; i < browsers.size (); ++i){echo "Testing␣the␣${ browsers[i]}␣browser"}

}}

}}

}

2.6.5 Other Declarative Pipeline options

A Declarative Pipeline can still use the more complex possibilities ofScripted Pipeline using a script step.

There are various other options that can be defined inside a declar-ative Jenkinsfile, but those are not relevant to the scope of this thesisas it is focused particularly on agents. These options can be found inthe Pipeline documentation page called Pipeline syntax.

21

2. Continuous Integration and Declarative Pipelines

2.6.6 Nomenclature

Over the years, Jenkins have introduced new nomenclature and de-finitions for its parts for various reasons. Some were technical, likeNode or Label, some were political, like Slave or Agent. In this shortsection, I described the names and their usages to eliminate confusionin following sections.

∙ Node - Used in Scripted Pipeline

∙ Agent - A server. Used since Jenkins version 2.x.

∙ Slave - Old name for a server. Same as agent. Used in Jenkinsversion 1.x.

∙ Label - It is a name. It could be used as a reference for any ofthe above.

2.7 Summary

This chapter provided a general insight into Continuous Integrationand its related concepts. It described Configuration as Code principle,which is the desired outcome of this thesis as a solution to a classicconfiguration. It also includes a basic description of Jenkins and ex-plained in detail those features, that are important for understandingthe solution design.

22

3 Cloud Computing

Cloud Computing refers to both the applications delivered as servicesover the Internet and the hardware and systems software in the datacenters that provide those services. The services themselves have longbeen referred to as Software as a Service (SaaS), so the term is used.The data center software and hardware is what is called a Cloud [17].

To offer a simpler explanation, Cary Landis and Dan Blacharskisay in their book Cloud Computing Made Easy [18] that in reality,the term computing encompasses other forms of computing beyondsoftware. In many ways, cloud computing is very similar to desktopcomputing in that it encompasses the same three basic elements:

∙ hardware (infrastructure),

∙ operating systems (platforms),

∙ software.

But the main difference is that cloud computing is really about rent-ing all the three elements over the Internet. On the other hand, withdesktop computing, they are managed locally.

3.1 Software as a Service

Service orientation is a trend in software engineering that promotesthe construction of applications based on entities called services. Thenotion of a service is, however, not concretely defined and can repre-sent different concepts to different stakeholders [19].

Service involves a provision of knowledge or skills from a person,team or organization which has for the benefit of another, often doneas a single, customized job. The output of a service is intangible. Ser-vice also is information and knowledge intensive, non-storable, lacksmobility and consumption runs simultaneously with the supply. Thelast characteristic of a service is especially important for this thesis.

23

3. Cloud Computing

3.1.1 Cloud as a service

In cloud terminology, the phrase “as a service” is extensively used,which simply means that a given cloud product (whether infrastruc-ture, platforms, or software) is offered in a way that it can be rented byconsumers over the Internet. By “rented,” we are implying that youpay only for what you use. It is often described as an “on demand”service because it is available whenever you need it [18].

3.2 Benefits of Cloud Platforms

Cary Landis and Dan Blacharski [18] also talk about benefits of CloudPlatforms:

∙ Lower costs - Because the non-core code is already engineered,cloud infrastructure can reduce costs by more than 80%.

∙ Lower risks – Likewise, a cloud platform can reduce risks by asmuch because common functions are already tested, sometimesover a period of years.

∙ Higher profit margins – Software developers and system inte-gration firms can deliver more for substantially less, thus highermargins on fixed price contracts.

∙ Rapid prototyping – Create and deploy concept applicationswithout writing code.

∙ Higher security and interoperability – The National Instituteof Standards and Technology says that there are major securityissues in the cloud caused by cloud providers implementationsof various unproven models of security. Cary Landis and DanBlacharski say that cloud platforms have a common, well-knownand test security model. If cloud software uses the platform,then it is secure.

3.3 Cloud elasticity

In article Cloud Elasticity Using Probabilistic Model Checking [20],Cloud elasticity is described as the degree to which a Cloud system is

24

3. Cloud Computing

adaptable to operation volume changes or requests by automaticallyprovisioning and deprovisioning resources, so that at any point in timethe provided resources match the current need as closely as possible.

Contemporary continuous data flow systems use elastic scaling ondistributed cloud resources to handle variable data rates and to meetapplications’ needs while attempting to maximize resource utilization[21].

In modern days, elasticity is a major component of cloud comput-ing. It saves customers money and provides high-cost efficiency. Itmeans that customers pay only for what they really need and that theyare not tied up to a concrete amount of computing power subscription.It allows customers to frequently demand, use, change and releaseany computing resources they need.

In the publication Elasticity in Cloud Computing: What It Is, andWhat It Is Not [22], authors describe two Dimensions and Core Aspectsof cloud elasticity:

∙ Speed - There are two types of scaling: up and down. The timeit takes to switch from an underprovisioned state to an optimalstate is the time of scaling up. The time it takes from overpro-visioned state to an optimal state is the speed of scaling down.But the speed of scaling up/down does not correspond directlyto the technical resource provisioning/deprovisioning time.

∙ Precision - The precision of scaling is defined as the absolutedeviation of the current amount of allocated resources from theactual resource demand.

3.3.1 Amazon EC2

In 2006 Amazon introduced Amazon Elastic Compute Cloud (AmazonEC2) as part of Amazon Web Services (AWS). EC2 was the first realavailable service for renting computing power for customers own useof software.

Amazon’s description of EC2 [23] gives an easy to understandinsight to elastic cloud computing: Amazon EC2 is a web service thatprovides resizable compute capacity in the cloud. It is designed tomake web-scale computing easier for developers. Just as Amazon Sim-ple Storage Service (Amazon S3) enabled storage in the cloud, Amazon

25

3. Cloud Computing

Figure 3.1: Traditional vs elastic cloud computing. Source: [24]

EC2 provided to “compute” in the cloud. It provided complete controlof computing resources. Amazon EC2 reduced the time required to ob-tain and boot new server instances to minutes, allowing customers toquickly scale capacity, both up and down, as the requirements change.

Amazon EC2 changed the way customers were paying for thecomputing power by just paying for what they were actually using[24].

The comparison in provisioning traditional Cloud and Elasticcloud can be seen in Figure 3.3.1.

3.4 OpenStack

OpenStack is open source software for building public and privateclouds [25]. OpenStack is one of the most popular open-source projectswith thousands of community members. It is backed by some of thebiggest organizations involved in Cloud and IT hosting.

OpenStack is a cloud operating system that controls large pools ofcompute, storage, and networking resources throughout a datacenter,all managed through a dashboard that gives administrators controlwhile empowering their users to provision resources through a webinterface [26].

On the official website of OpenStack is a graph visually explainingwhat OpenStack is. It can be seen in Figure 3.2.

26

3. Cloud Computing

Figure 3.2: What is OpenStack. Source: [26]

3.5 Summary

In this chapter, a theory behind Cloud computing was explained. Itdescribed cloud elasticity and its brief history. OpenStack was intro-duced but did not require as much insight as Jenkins in the previouschapter because Jenkins is the part which is going to be extended withalready provided OpenStack solution in the plugin.

27

4 Problem statement

This chapter provides the analysis of the problem and the proposedsolution. The implementation request was created by my Red Hat tutor,Oliver Gondža. It was done by creating a Jira issue called Introducedeclarative pipeline agent support. The Jira description was: Providean alternative way to provision Jenkins agent from OpenStack throughopenstack-cloud-plugin through declarative pipeline agent statement.

4.1 Problem description

Because Jenkins Declarative Pipeline is a popular feature among Jenk-ins users, it is important to slowly start implementing its functionsinto other plugins also. But with introducing the Declarative Pipelineapproach, the existing functionality of the OpenStack Cloud pluginhad to be slightly altered. This lead to the second goal of the thesis: tolet users define their own agents.

The OpenStack Cloud Plugin, as it had been before this thesis, hadnot let users create their own agents. It had to be created by Jenkinsadministrator and put into a template. A template could contain oneor more agents. So if a user or developer wanted a special machine torun his or her build, a request for the administrator to create it had tobe done. This was time-consuming and required a new design with amore modern approach.

So there were two goals for the implementation part:

1. Implement Declarative Pipeline feature

2. Let the user define their own agent

4.2 Solution design

4.2.1 Learn the theory

To begin working on the solution I had to learn the theory first. I havebeen in an automation profession for some time so I was familiar withJenkins and the theory behind automation but I had never workedwith OpenStack or dealt with Cloud in any particular way. It was also

29

4. Problem statement

Figure 4.1: Components diagram

necessary to study extending of Jenkins because even though I wasan experienced Jenkins user, I have never worked on a Jenkins plugindevelopment before.

By the requirement of my tutor, I was asked to learn the basicstructure of OpenStack Cloud Plugin, try to build it and look into itstests.

So the first phase of preparation was about studying theory andan existing code base. These were the main goals:

1. Learn about OpenStack and a basic theory behind Cloud

2. Learn about Jenkins plugin development

3. Study the existing OpenStack Cloud Plugin code, tests and buildthe plugin.

4.2.2 Start implementing

To start an actual implementation, I had to find out the starting point.After studying Jenkins plugin development I learned that I need tofind necessary extension points. I studied existing Pipeline pluginsand learned that I have to start with Pipeline Model Definition Plugin.The second important part was to implement a step that would createthe agent in OpenStack. The design of components that were going tobe tied together after implementation are showed in Figure 4.1.

30

4. Problem statement

Figure 4.2: Sequence diagram

Jenkins will have much more plugins installed but for the sake ofsimplicity and explaining the design, these are the most important.OpenStack Cloud Plugin needs to extend abstract classes from PipelineModel Definition Plugin to allow Declarative Pipeline openstack agentand Workflow Step API Plugin to define a Step in which an OpenStackagent will be created. OpenStack Cloud Plugin then receives necessarydata from a Jenkinsfile and requests agent provision from a providedOpenStack instance.

To provide a different explanation, I have also created a sequencediagram with more details. It can be seen in Figure 4.2.

31

5 Development

This chapter covers the development process. It is not written inchronological order of how it was really implemented to maximizereadability and intelligibility of the text. During the developmentprocess, many issues and bugs were encountered. The most major andrelevant problems are explained in this chapter. In the end, there isan example of a successful run in a real instance of OpenStack. Codeexamples are used for better understanding.

5.1 Initial preparation

The first requirement to start working on the issue was to understandthe existing code bases of multiple plugin projects. I had to get familiarwith the most important plugin, which is the OpenStack Cloud Pluginfor Jenkins. This plugin started as a fork of JClouds Cloud Plugin in2015 to serve solely an OpenStack purpose. It was trimmed down butstill uses its nomenclature in its class names.

I also studied documentation of Jenkins and Jenkins extendingtutorials. I learned about standard Jenkins libraries, Extension Points,Apache Jelly, Maven HPI plugin or Descriptors.

Inspection and studying of Pipeline syntax to determine the re-quired additions to its DSL and to define basic requirements wasnecessary to start developing. Fortunately, the syntax documentationis well written and easy to learn.

5.2 Maven configuration

Before implementing any code, Maven’s pom.xml configuration fileneeded to be updated. OpenStack Cloud Plugin, as it was when thisthesis implementation started, used Jenkins version 1.642.3. This ver-sion was from Match 2016. Because that was Jenkins version 1.x, itdid not support Jenkins Pipelines. It had to be updated to newer aLong Term Support version. A version 2.60.3 was chosen, which isfrom August 2017.

33

5. Development

Updating Jenkins version meant also a necessary upgrade to exist-ing Maven dependencies. There were 13 Maven dependencies, fromwhich 9 needed a version upgrade.

5.2.1 Adding dependencies

The plugin that defines Jenkins Pipeline agents is called PipelineModel Definition Plugin. It has two modules important for this thesis’spurpose: pipeline-model-definition and pipeline-model-exten-sions. The first module defines existing agents and their executionsteps. The second module implements an abstract class Declarative-Agent which is the first class that needed to be extended in order toprovide an option to have an OpenStack agent.

This is the final list of added dependencies with short descriptions:

1. Pipeline Model Definition Plugin

2. Structs Plugin - Library plugin for DSL plugins that need con-cise names for Jenkins extensions.

3. Pipeline Step API Plugin - Provides Pipeline Step extendingoptions.

4. Pipeline API Plugin - Defines Pipeline API.

5. Variant plugin - Allows Jenkins plugins to behave differentlybased on the environment it is running in.

6. Script Security Plugin - Allows Jenkins administrators to con-trol what in-process scripts can be run by less-privileged users.

7. SCM API Plugin - Provides a new enhanced API for interactingwith SCM systems.

8. Apache Commons Codec - Simple encoder and decoders forvarious formats such as Base64 and Hexadecimal.

34

5. Development

Figure 5.1: Check if a Declarative Pipeline functionality

5.3 Test-driven development

Because it was decided that this feature is going to be implementedusing test-driven development methodology, adequate tests neededto be prepared.

The tests were implemented using already provisioned tests in aclass PluginTestRule, which extends Jenkins’s TestRule class.

The first test tests accepting of an OpenStack agent. The require-ment was to create a Jenkins job run with an agent openstack field.The test does not verify OpenStack machine creation, only that the jobhas run. This was mostly important during first stages of implemen-tation when OpenStack connection was not provided and job, stepand node creations needed to be tested. It asserts a "Hello World!"output. The test is shown in Figure 5.1.

The second test mocks OpenStack machine provisioning. It ver-ifies that there is only one machine running and tests of Jenkinsfileconfiguration was propagated into an OpenStack request.. The secondtest is shown in Figure 5.2.

5.4 Implementing an OpenStack agent

Providing a new agent is done via abstract class DeclarativeAgent ofPipeline Model Definition Plugin. A new DeclarativeOpenstackAgentclass was created which extends said abstract class.

There are two ways of setting optional and mandatory fields in Jenk-ins objects. First, is using an annotation @DataBoundSetter on a setter

35

5. Development

Figure 5.2: Mock and check of a new machine creation

method of a field. Second is using an annotation @DataBoundConstru-ctor on a constructor. For the purposes of declarative OpenStackagent, a data-bound constructor was used. It defines every field that ismandatory and thus needs to be put into a Jenkinsfile section: agent:openstack. Into this constructor, all the parameters required to provi-sion a new OpenStack agent are passed. How the parameters lookedin the final version, with an agent: openstack can be seen in an Ap-pendix A and Appendix B.

5.4.1 Agent options

In this subsection, all the available and mandatory parameters arebriefly explained.

∙ Cloud - A name for preconfigured connection to an OpenStackinstance.

∙ BootSource - Saved preinstalled operating system. OpenStacksupports multiple formats. From classic virtualization as qcow(snapshot) to hard-disc image with a filesystem (volume).

∙ hardwareId - Also called a flavor. It is a combination of a numberof virtual CPUs, RAM and discs.

36

5. Development

∙ networkId - A virtual network to which a machine is about tobe connected.

∙ userDataId - An ID of a preconfigured file in Jenkins, whichconsists of post-installation instructions.

∙ floating ip pool - Google

∙ securityGroups - A name of a group of security network rulesin OpenStack.

∙ availabilityZone - A geographical destination

∙ keyPairName - A preconfigured name of a pair of private andpublic keys to access a server.

∙ startTimeout - An amount of time that OpenStack Cloud Pluginwaits until the agent is provisioned.

∙ JVM options - Another parameters for agent process.

∙ fsRoot - Agent working directory.

∙ launcherFactory - A type of communication with an agent. Canbe JNLP or SSH.

5.5 A Pipeline Step

In order to execute new implementations in Pipeline, a Pipeline Stepneeds to be extended. A new class OpenStackNodeStep was created.Its purpose is to instantiate new SlaveOptions object. SlaveOptionsis a class, from which OpenStack gets all necessary information tocreate a new machine. So in order to create a new agent, an instanceof this class is needed.

OpenStackNodeStep has a @DataBoundConstructor, which is usedto instantiate the class from the provided fields in Jenkinsfile (how itis done is explained in Section 5.6).

37

5. Development

5.5.1 Changes in the original code

Before the implementation of this thesis, provisioning of OpenStackagents had only been possible by utilizing templates. These templatescontain the necessary information to create an agent, which is regularlyused. Part of these data are slave options.

Because the purpose of this thesis is to provide one-time use agentsdeclared by a user, templates are not necessary. But it was not possibleto instantiate SlaveOptions class without using a template. Becauseof that, a new class TemporaryServer was created by reusing the codefrom the templates. This code was then altered to allow the creationof OpenStack agents directly, only by using SlaveOptions data.

OpenStack Cloud

It is worthy to note that TemporaryServer class also connects a to-be-created agent to an already existing Cloud. The name of the cloudneeds to be declared in Jenkinsfile as one of the parameters. Cloudsare typically created by an administrator.

5.5.2 Step Execution

In order to trigger any Step, it needs to have an Execution class that isinstantiated inside of the Step itself.

OpenStackNodeStepExecution class then overrides a startmethod,in which a required Cloud is found, a TemporaryServer instantiatedwith proper SlaveOptions instance, a new JCloudsSlave provisionedinside OpenStack and triggered. The constructor of this class with apreviously mentioned sequence of commands inside its start methodcan be seen in Figure 5.3. Step Executions can have several implemen-tations. Based on GitHub documentation, SynchronousNonBlocking-StepExecution class was extended because it is used for a generalpurpose and also recommended by the documentation for specificuse cases that can survive Jenkins restarts.

This is the final part of an OpenStack slave creation flow. Afterward,the execution stages of the Jenkinsfile are executed on the providedslave.

38

5. Development

Figure 5.3: OpenStackNodeStepExecution

Leaving out some parameters

A createSlaveOptions method needed to be created because not allthe SlaveOptions fields are required for the purposes of a temporaryserver for a Declarative Pipeline. Fields instanceCap, numExecutorsand retentionTime do not make sense to be defined by a user, henceare set to default values. It is not possible to define these fields insidea Jenkinsfile. In order to make their definition possible (and also op-tional), they would need to be defined in DeclarativeOpenStackAgentclass with a @DataBoundSetter annotation as it is explained in Section5.4.

5.6 Groovy script

In order to create a Pipeline Step, explained in the previous Section5.5, a new Groovy class DeclarativeOpenstackAgentScript, whichextends an abstract class DeclarativeAgentScript was created. Itstype is DeclarativeOpenstackAgent from which it uses its fields sothey could be passed to a OpenStackNodeStep constructor. This is

39

5. Development

Figure 5.4: Passing of OpenStack arguments from Jenkinsfile to theStep

easily done by a method getAsArgs, that puts all arguments fromprovided Jenkinsfile into a TreeMap.

DeclarativeOpenstackAgentScriptneeds to override runmethodfrom its parent. Inside this method, arguments from the agent getpassed to a Step in a try-catch-finally block.

5.7 Class diagram

As a final explanation of development process, the code is wrapped upin a simple class diagram. This diagram was created for the purposeof understanding how the mentioned classes work and communicatetogether, so many unimportant data are left out. The class diagram isavailable in Appendix C. Most of the classes I created are included inthis diagram and has a red border. Those that are from other Jenkinsplugins are green and SlaveOptions is yellow because it had alreadybeen implemented before. There were other extended or implementedclasses but would make the diagram only less simple. The wholecode can be seen for example in the GitHub pull request 203: https://github.com/jenkinsci/openstack-cloud-plugin/pull/203.

40

5. Development

5.8 Encountered issues

5.8.1 Names of plugins

During a research part, at the beginning of the work, it was difficultto differentiate plugins or connect names together. There are threeplaces to look up information about Jenkins plugins: wiki.jenkins.io,plugins.jenkins.io and github.com. For example, Pipeline pluginhas these names in the same order as mentioned websites: Pipeline Plu-gin, Pipeline and workflow-aggregator-plugin. Especially the Pipelinename gets confusing.

There was also a problem with different names for servers. This isexplained in Section 2.6.6.

5.8.2 Poor documentation of many plugins

Some Jenkins plugins don’t have any kind of documentation. Javadocis usually missing, which is unfortunate for a plugin meant to beextended.

In case of Pipeline Model Definition Plugin, I had to look up Ex-tension points of all Jenkins plugins to find a right place to start im-plementing a new agent. I had to learn the functionality of multipleplugins which I needed to understand what they do. There is alsolittle documentation on how to extend Jenkins Pipeline even thoughthere is plenty about extending Jenkins itself.

I had to communicate with developers via mailing lists and IRCchannels of Jenkins to understand some parts of Jenkins Pipeline.

5.8.3 Declarative Pipeline DSL

Even though the Domain Specific Language is easy to read and un-derstand, it was difficult to find out how to instantiate a new objectinside a Jenkinsfile. The approach seen in Appendinx A is a result ofnumerous tries of different solutions we came up with. Most of thetime, I was using a solution of parsing strings and creating the objectsinside Java classes.

41

5. Development

5.8.4 Wrong version of Jenkins

From the very beginning, it was clear that the Jenkins version in Open-Stack Cloud Plugin needed to be upgraded. When the work on thesolution of this thesis started, the plugin was constrained to Java ver-sion 7. Because of that, a Jenkins version which had Java 7 supportand Pipeline support was chosen. It was a version 2.7.3 which provedto be filled with issues. Even though all the provisioning tests passed,the feature was not possible to run in production and would have towait for the OpenStack Cloud Plugin to update its version of Jenkinsand mainly Java 8 support.

Fortunately, during the implementation of this thesis, the masterbranch of the plugin received support for Java 8. There still wasn’t anyJenkins version upgrade, but it was a sufficient foundation for thiswork. The code was rebased and during a difficult process of resolvingall the Maven dependencies, it was possible to upgrade the versionof Jenkins to 2.60.3 which do not have blocker type of bugs for thisfeature.

5.9 Production testing

It was necessary for the feature to be tested on a real OpenStack. Thiswould be difficult for an individual without an access to an OpenStackinstance. But because Red Hat uses OpenStack for its own purposesand has many instances, I was able to use one for development pur-poses.

Running OpenStack instance is only a part of the solution becausea running instance of Jenkins with the actual plugin also needs to beprovided. That can be done using several solutions. These two wereused during the final phase of development and testing of this thesis:plugin development testing environment using Maven HPI Pluginand running a standalone Jenkins WAR deployment.

5.9.1 Environment for plugin development testing

Maven HPI plugin is a helpful tool when it comes to testing Jenkinsplugins in development. If the plugin is in a buildable state, runninga command mvn hpi:run builds and runs an instance of Jenkins on

42

5. Development

Figure 5.5: Directly installing the plugin into Jenkins

a localhost. This instance has the plugin already installed with onlynecessary dependencies. This is very close to a real production run.

5.9.2 Standalone Jenkins deployment

Another option is to download a relevant Jenkins WAR file (a versionsame as defined in the plugin) from the official repository, create atemporary JENKINS_HOME and run the WAR file. This will run aclean Jenkins instance on a localhost. When opening up Jenkins forthe first time, the user is asked to set up plugins installations. Thedefault set of plugins were picked in case of this thesis. Then, usingdirect upload, the OpenStack Cloud Plugin was installed. The directinstall is shown in Figure 2.3 and in Jenkins can be found in Advancedoptions for plugin management of Jenkins UI.

5.9.3 Setting up an OpenStack connection

At first, an administrator has to connect Jenkins to a Cloud in Open-Stack. This is done in Manage Jenkins -> Configure System configura-tion page. Cloud Name, End Point URL and Credentials to log in tothe OpenStack instance are required.

For the purposes of SSH connection, SSH credentials have to be setup inside Jenkins credentials configuration. An OpenStack private keyneeds to be provided. Jenkins automatically generates an ID for thesecredentials. This ID is then put into Jenkinsfile as LauncherFactory.

43

5. Development

Figure 5.6: Configuration used for a real build

5.9.4 Setting up a job

The type of a job required for this feature is a Pipeline job. In thesettings of the Job, the most fundamental requirement is to provide aDeclarative Jenkinsfile. It can be either by a link to a remote repositoryor defined as a script inside the UI. An example of the second approachis shown in Figure 5.6.

5.9.5 Running a build

After triggering the build, a log containing something similar like inFigure 5.7 will be shown.

In OpenStack, a temporary server was created - Figure 5.9. Aftersuccessful creation in OpenStack, it should appear in Jenkins like inFigure 5.8.

44

5. Development

Figure 5.7: Successful build in Jenkins

45

5. Development

Figure 5.8: Temporary server available in Jenkins

Figure 5.9: Agent created in Red Hat’s instance of OpenStack

46

5. Development

5.10 Summary

This chapter covered the implementation process. The most importantpieces of code were shown on figures and then summarized on aclass diagram. The implementation was not smooth due to lack ofdocumentation and some other problems were encountered. Thoseissues were explained at the end of the chapter. The solution wassuccessful and is working.

47

6 Conclusion

The goal of this thesis was to implement Declarative Pipeline featurefor OpenStack Cloud Plugin of Jenkins and provide a basic theoryexplanation about relevant topics.

The most important theory part was about automation and Jenkinsitself. Because both topics are too big to explain in detail, only themost relevant principles were explained in more depth. The thesisalso explains what is Configuration as Code methodology and itsbenefits.

In Problem statement chapter, the thesis explains the process ofdesigning a solution to the problem. It also provides diagrams tounderstand how the problem is going to be implemented in nextchapter.

In Development chapter, a step by step implementation of the issueis provided. At the end, it explains encountered problems. It used anon-chronological system of explanation to better understand thedesign.

The feature was tested both in a testing environment and on areal instance of Jenkins. Because OpenStack is an enterprise-focusedsoftware, the feature was tested on an existing and running instanceof OpenStack by Red Hat. This instance is used for building of Key-cloak project. The OpenStack instance is basic and not altered besidesgraphical user interface template. All the tests have passed and thefeature is ready to be merged.

The feature was wrapped in a single commit with the number andthe name of the Jira issue. It is waiting for merge and will be includedin next alpha release as a planned feature in next release. BecauseOpenStack Cloud Plugin is used by big organizations such as Red Hat,the feature needs time to be properly tested by plugins users before itcan be fully merged into a proper release.

Because this thesis impacted the version of Jenkins used in Open-Stack Cloud Plugin, we checked the plugin usage statistics which showthat 81% of users won’t be impacted by the version change as theyalready use version 2.60.3 or above. 12% of users will be affected by thechange and if they want to use the feature, an upgrade of their Jenkins

49

6. Conclusion

instance will be needed. At last 7% of users were not even using thelatest version of OpenStack Cloud Plugin with Jenkins version 1.651.

50

Bibliography

1. SELIC, B. The pragmatics of model-driven development. IEEE Software.2003, vol. 20, no. 5. ISSN 0740-7459.

2. PAUL GRÜNBACHER, Yves Ledru. Automated Software Engineering -Introduction. 2004. Available also from: https://www.ercim.eu/publication/Ercim_News/enw58/intro.html.

3. FUGGETTA, Alfonso; DI NITTO, Elisabetta. Software Process: Pro-ceedings of the on Future of Software Engineering. 2014. ISSN 978-1-4503-2865-4. Available also from: http://doi.acm.org/10.1145/2593882.2593883.

4. BERTRAM, Adam. Infrastructure as code benefits range from audits to ops.2017. Available also from: https://searchitoperations.techtarget.com/tip/Infrastructure-as-code-benefits-those-willing-to-learn.

5. SITAKANGE, Jafari. Infrastructure as Code: A Reason to Smile. 2014.Available also from: https://www.thoughtworks.com/insights/blog/infrastructure-code-reason-smile.

6. TANG, Chunqiang; KOOBURAT, Thawan; VENKATACHALAM, Pradeep;CHANDER, Akshay; WEN, Zhe; NARAYANAN, Aravind; DOW-ELL, Patrick; KARL, Robert. Holistic configuration managementat Facebook: Proceedings of the 25th Symposium on OperatingSystems Principles. 2015.

7. MORRIS, Kief. Infrastructure as Code: Managing Servers in the Cloud.O’Reilly Media, 2016. ISBN 1491924357.

8. FOWLER, Martin. Continuous Integration. 2006. Available also from:https://martinfowler.com/articles/continuousIntegration.html.

9. SMART, John Ferguson. Jenkins: The Definitive Guide: Continuous Inte-gration for the Masses. O’Reilly Media, 2011. ISBN 1449305350.

10. DUVALL, Paul M. Continuous Integration: Improving Software Qual-ity and Reducing Risk. Addison-Wesley Professional, 2007. ISBN9780321336385. Available also from: https://www.xarg.org/ref/a/0321336380/.

51

BIBLIOGRAPHY

11. CHEN, L. Continuous Delivery: Huge Benefits, but Challenges Too.IEEE Software. 2015, vol. 32, no. 2. ISSN 0740-7459.

12. HUMBLE, Jez. Continuous Delivery: Reliable Software Releases throughBuild, Test, and Deployment Automation (Addison-Wesley Signature Se-ries (Fowler)). Addison-Wesley Professional, 2010. ISBN 0321601912.

13. DANCIU, Dan. Continuous Delivery. 2014. Available also from: https://www.todaysoftmag.com/article/1068/continuous-delivery.

14. BOOCH, Grady. Object-Oriented Analysis and Design with Applications(2nd Edition). Addison-Wesley Professional, 1993. ISBN 0805353402.Available also from: https://www.xarg.org/ref/a/0805353402/.

15. PHILLIPS, Andrew. The Continuous Delivery Pipeline - What it is andWhy it’s so Important in Developing Software. 2014. Available alsofrom: https://devops.com/continuous-delivery-pipeline/.

16. JENKINS. Pipeline Syntax. 2018. Available also from: https://jenkins.io/doc/book/pipeline/syntax/.

17. FOX, Armando; GRIFFITH, Rean; JOSEPH, Anthony; KATZ, Randy;KONWINSKI, Andrew; LEE, Gunho; PATTERSON, David; RABKIN,Ariel; STOICA, Ion. Above the clouds: A Berkeley view of cloudcomputing. Dept. Electrical Eng. and Comput. Sciences, University ofCalifornia, Berkeley, Rep. UCB/EECS. 2009, vol. 28, no. 13.

18. CARY LANDIS, Dan Blacharski. Cloud Computing Made Easy: An Easyto Understand Reference About Cloud Computing. CreateSpace Inde-pendent Publishing Platform, 2013. ISBN 1482779420. Availablealso from: https://www.xarg.org/ref/a/1482779420/.

19. STOJANOVIC, Zoran. Service-Oriented Software System Engineering:Challenges and Practices. IGI Global, 2004. ISBN 1591404266. Avail-able also from: https://www.xarg.org/ref/a/1591404266/.

20. NASKOS, Athanasios; STACHTIARI, Emmanouela; GOUNARIS, Anas-tasios; KATSAROS, Panagiotis; TSOUMAKOS, Dimitrios; KON-STANTINOU, Ioannis; SIOUTAS, Spyros. Cloud elasticity usingprobabilistic model checking. arXiv preprint arXiv:1405.4699. 2014.

21. KUMBHARE, Alok; SIMMHAN, Yogesh; PRASANNA, Viktor K. Ex-ploiting application dynamism and cloud elasticity for continuousdataflows: High Performance Computing, Networking, Storage andAnalysis (SC), 2013 International Conference for. 2013.

52

BIBLIOGRAPHY

22. HERBST, Nikolas Roman; KOUNEV, Samuel; REUSSNER, Ralf H.Elasticity in Cloud Computing: What It Is, and What It Is Not. 2013,vol. 13.

23. AMAZON. Announcing Amazon Elastic Compute Cloud (Amazon EC2)- beta. 2006. Available also from: https://aws.amazon.com/about-aws / whats - new / 2006 / 08 / 24 / announcing - amazon - elastic -compute-cloud-amazon-ec2---beta/.

24. GALANTE, Guilherme; BONA, Luis Carlos E. de. A Survey on CloudComputing Elasticity: Proceedings of the 2012 IEEE/ACM FifthInternational Conference on Utility and Cloud Computing. IEEEComputer Society. 2012. ISSN 978-0-7695-4862-3. Available also from:http://dx.doi.org/10.1109/UCC.2012.30.

25. JACKSON, Kevin. OpenStack Cloud Computing Cookbook - Third Edition.Packt Publishing - ebooks Account, 2015. ISBN 1782174788.

26. OPENSTACK. What is OpenStack? 2018. Available also from: https://www.openstack.org/software/.

53

A Final version of test Jenkinsfile

55

B OpenStack Declarative Agent

57

C Class diagram

59