Climbing Mont Blanc and Scalability

Christian Chavez

Master of Science in Computer Science

Supervisor: Lasse Natvig, IDI

Department of Computer and Information Science

Submission date: July 2016

Norwegian University of Science and Technology

Problem StatementClimbing Mont Blanc and ScalabilityClimbing Mont Blanc (CMB) is a system for evaluation of programs executed on modernheterogeneous multi-cores such as the Exynos Octa chips used in, e.g., Samsung GalaxyS5 and S6 mobile phones, see https://www.ntnu.edu/idi/card/cmb. CMB evaluatesboth performance and energy efficiency and provides the possibility of performance rank-ing lists and online competitions. A first version of the system is available and undertrial use. This master thesis project is focused on improving the system with increasedscalability so that the system can handle more user submissions per hour.

The project involves the following subtasks:1. Study the existing solution in CMB for automatic system monitoring and recovery

and suggest improvements.2. Describe and implement a dispatcher in the current CMB system that will allow

the use of multiple XU3 backends to serve concurrent user submissions.3. Test the dispatcher with two or three XU3 boards. Implement a simple script for

generating a synthetic load (simulating users and submission) to be able to evaluatethe scalability of a new CMB variant using the developed dispatcher.

4. Describe how the dispatcher can be used to allow different boards than the XU3to be used together with XU3 boards. Discuss what effects this will have on otherparts of the system.

If time permits:5. Propose possible throttling techniques for cases where CMB gets too many/frequent

submissions from a single user, or in total from all active users.6. Propose compilation/Makefile improvements, and the possibility of a makefile that

can be edited by problem-setter through admin-interface.7. Propose how more detailed statistics such as performance counter values can be

given as feedback to the CMB user.8. Propose the addition of more programming languages and libraries.9. Propose an extension of the dispatcher into a load-balancing broker.

10. Implement some of the proposed solutions after approval by, and in collaborationwith the CMB team.

The master thesis project is part of the EECS Strategic Research project at IME (www.ntnu.edu/ime/eecs).

Supervisor: Prof. Lasse Natvig

i

Abstract

This thesis details a proposed system implementationupgrade for the CMB system, accessible at climb.idi.ntnu.no, which profiles C/C++ code for its en-ergy efficiency on an Odroid-XU3 board, which utilisesa Samsung Exynos 5 Octa CPU, and has an ARMMali-T628 GPU. Our proposed system implementa-tion improves the robustness of the code base andits execution, in addition to permitting an increasedthroughput of submissions profiled by the system withthe implementation’s dispatcher which allows the sys-tem to utilise several Odroid-XU3 backends for theenergy and timing measurement profiling. Our testsshow that our implementation can achieve more than4x speedup with “Hello World” submissions using aparallelized web server, and around 2x speedup with“Shortest Path” submissions using a serial web server.

ii

Preface

I have many I want to acknowledge for all their help, support, and comradeship duringmy time in academia (not just for the duration of this master project, at the end of theline), but individuals of note include the following:

• Lasse Natvig, my supervisor for this Master project, for his patience, wisdom, andguidance through the course of this project.

• IDI’s Technical group, especially Arne Dag Fidjestøl, Jan Grønsberg, and ErikHoumb, for their support and tips in the development of this project’s proposedsystem implementation.

• Sindre Magnussen, for his help in understanding and learning the workings of theCMB system.

• Dag Frode Solberg, Christoffer Viken, (and the rest of the crowd from NTNU’sPVV), for all those hours, spent as my rubber ducks and coming with tips andguidance when my technical competence was insufficient.

• Finn Inderhaug Holme, for saving my bacon when the power connections of theequipment in Trondheim had to be disconnected and reconnected after I had movedto Oslo due to the delays incurred during this project.

• My family for their support and love throughout.

• And anyone else whose notable assistance may have (temporarily!) been forgottenduring the time of this writing.

Note: This report is rather long and was not condensed down as is the norm, due to thetime constraints described on the next page. However, I want to make it clear thatif you care about trees, you should not print this report in its entirety. Chapters 5,6, and Appendixes A, C, D, are all rather long, and especially boring to read onpaper.

iii

The main challenge faced during this projectAt the outset of this 21-week, contractually decided period allotted for this master project,Celery was chosen by the author of this project with the support of the supervisor, LasseNatvig, to be a promising and efficient way to solve a majority of the challenges of thisproject.

However, on the 15th of March, in a meeting with a member of the institutes’s (IDI’s)Technical Group (Arne Dag Fidjestøl), and the other master student currently workingon the CMB project (Sindre Magnussen), it was concluded that Celery, while fit for thetask, was introducing more complexity into the system, than what was currently needed.This conclusion was based on the fact that the earlier project future use estimates of theCMB system were too ambitious, and thus the CMB system did not need to support sopotentially high frequencies and concurrently submitted submissions by its user base.

Therefore, in week 10 of this project (start of the project’s 21-week allotted time was the11th of January), it was decided that a proposed implementation based upon the use ofCelery, would be of little use to future iterations of the CMB system, at this time. Thus,the author of this project has reversed all efforts of implementing the use of Celery intoCMB and has instead landed upon (and implemented) the proposed system implementa-tion described in Chapter 4.

In addition to compensation for a three week documented sick-leave, this project hasreceived an extension of two additional weeks to compensate for this setback. On top ofall that, the author of this paper had to move residence from Trondheim to Oslo in thelast four weeks of this project, as the move had been planned and scheduled before thedelays happened and the compensation was given.

The author of this paper has had to omit goals and desired implementations/improve-ments of this project (and paper research), due to this time-constraining setback, butwants to state that had this project been granted 4-6 more weeks, the complete (or near-complete) implementation of both the database, and automatic system monitoring andrecovery (described in Chapters 4 and 9) may well have been realized.

Abbreviations and Glossary

ARM = Advanced RISC Machine (http://www.arm.com/)BSC = Barcelona Supercomputing CenterCMB = Climbing Mont Blanc https://climb.idi.ntnu.no/HPC = High-Performance ComputingMB = Mont-Blanc (The EU Project: https://www.montblanc-project.eu/)SoC = System(s)-on-ChipVM = Virtual Machine

iv

Table of Contents

Problem Statement i

Abstract ii

Preface iii

Abbreviations and Glossary iv

Table of Contents v

List of Tables ix

List of Figures x

List of Listings 1

1 Introduction 31.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Project Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.1 Automatic System Monitoring and Recovery . . . . . . . . . . . . 51.2.2 The Dispatcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Background 72.1 Mont-Blanc, The EU Project . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 The Climbing Mont Blanc System . . . . . . . . . . . . . . . . . . . . . . 82.3 Odroid-XU3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4 Concurrency Softwares Considered . . . . . . . . . . . . . . . . . . . . . . 9

2.4.1 ZeroMQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.4.2 Celery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

v

3 Related Work 153.1 Online Judges - Websites for Competitive Programming . . . . . . . . . . 153.2 Backend Parallelization Projects . . . . . . . . . . . . . . . . . . . . . . . 16

4 Proposed System Solution 194.1 The outset state of CMB . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1.1 Proposed solution to environment variables/settings . . . . . . . . 224.2 Automatic system monitoring and recovery . . . . . . . . . . . . . . . . . 234.3 Upgrade from Python 2.7 to 3.4 . . . . . . . . . . . . . . . . . . . . . . . . 26

4.3.1 Git submodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.4 Database changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.4.1 Changes necessitated by the Dispatcher . . . . . . . . . . . . . . . 284.4.2 Potential changes for software language support . . . . . . . . . . . 29

4.5 The Dispatcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.6 The Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5 Server Installation Instructions 355.1 Getting the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365.2 Install Instructions and Pre-Requisites . . . . . . . . . . . . . . . . . . . . 375.3 Starting the Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5.3.1 Start-up Script Differences . . . . . . . . . . . . . . . . . . . . . . 39

6 Backend Installation Instructions 406.1 Install Instructions and Pre-Requisites . . . . . . . . . . . . . . . . . . . . 416.2 Getting the code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446.3 Starting the Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7 Methodology 477.1 Hardware & Hardware Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.1.1 CMB Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487.1.2 Backends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7.2 Software & Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . 507.2.1 CMB Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517.2.2 Backends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.3 Upload- and Profiling- test-problems . . . . . . . . . . . . . . . . . . . . . 527.4 Benchmark Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

7.4.1 Benchmark Tests set-up . . . . . . . . . . . . . . . . . . . . . . . . 547.4.2 Challenge due to timing difference between backends . . . . . . . . 54

7.5 Parallelization Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557.5.1 Parallelization Tests set-up . . . . . . . . . . . . . . . . . . . . . . 55

8 Results 578.1 Benchmark Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578.2 Parallelization Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8.3.1 Benchmark Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

vi

8.3.2 Serial Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648.3.3 Parallel Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 668.3.4 Parallel Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

9 Future Work 719.1 Completing the Automatic System Monitoring and Recovery implementation 719.2 Future improvements to the Dispatcher . . . . . . . . . . . . . . . . . . . 74

9.2.1 Expanding the dispatcher into a broker . . . . . . . . . . . . . . . 749.2.2 Discovering the upper limit of backends a server can handle . . . . 759.2.3 Fixing the undiscovered Gunicorn bug . . . . . . . . . . . . . . . . 77

9.3 Expanding CMB to support language-specific problems/submissions . . . 799.3.1 Permitting problem creators to edit C/C++ Makefile . . . . . . . 79

9.4 Remaining future potential improvements . . . . . . . . . . . . . . . . . . 809.4.1 Folder re-structuring . . . . . . . . . . . . . . . . . . . . . . . . . . 809.4.2 Adding new architectures/backends to the proposed system imple-

mentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 809.4.3 Improving and completing the DB schema in a future-proofing manner 819.4.4 Combining the efforts of Sindre Magnussen and this project . . . . 819.4.5 Stabilizing time requirements of the CMB software . . . . . . . . . 829.4.6 Improving server storage efficiency . . . . . . . . . . . . . . . . . . 839.4.7 Coverage testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

10 Conclusion and contribution 8510.1 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Bibliography 87

Appendices 91

A Test Set-Up Configs 92A.1 Source Scripts Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

A.1.1 Server Source Script File . . . . . . . . . . . . . . . . . . . . . . . . 93A.1.2 Backends Source Script File . . . . . . . . . . . . . . . . . . . . . . 94

A.2 Secret Environment Variable(s) Config Files . . . . . . . . . . . . . . . . . 95A.2.1 Server Secrets Config File . . . . . . . . . . . . . . . . . . . . . . . 95A.2.2 Backends Secrets Config File . . . . . . . . . . . . . . . . . . . . . 95

A.3 Machine-Specific Environment Variable(s) Config Files . . . . . . . . . . . 96A.3.1 Server Specific Config File . . . . . . . . . . . . . . . . . . . . . . . 96A.3.2 Backends Specific Config File . . . . . . . . . . . . . . . . . . . . . 97

A.4 Test-Server Gunicorn start-script/config . . . . . . . . . . . . . . . . . . . 97

B SSH Install and Set-Up Note 99

C Test-VM Specifications 101C.1 OS and Kernel Information . . . . . . . . . . . . . . . . . . . . . . . . . . 101C.2 CPU Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102C.3 Memory Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

vii

D Backends Specifications 105D.1 OS and Kernel Specifications of Backends . . . . . . . . . . . . . . . . . . 105

D.1.1 Backend 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105D.1.2 Backend 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106D.1.3 Backend 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

D.2 CPU Specifications of Backends . . . . . . . . . . . . . . . . . . . . . . . . 106D.2.1 Backend 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106D.2.2 Backend 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107D.2.3 Backend 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

D.3 Memory Specifications of Backends . . . . . . . . . . . . . . . . . . . . . . 109D.3.1 Backend 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109D.3.2 Backend 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111D.3.3 Backend 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

D.4 Packages installed on all three backends . . . . . . . . . . . . . . . . . . . 113D.5 Packages installed on backend 1 and not 2 . . . . . . . . . . . . . . . . . . 125D.6 Packages installed on backend 1 and not 3 . . . . . . . . . . . . . . . . . . 129D.7 Packages installed on backend 2 and not 1 . . . . . . . . . . . . . . . . . . 143D.8 Packages installed on backend 2 and not 3 . . . . . . . . . . . . . . . . . . 144D.9 Packages installed on backend 3 and not 1 . . . . . . . . . . . . . . . . . . 155D.10 Packages installed on backend 3 and not 2 . . . . . . . . . . . . . . . . . . 163

viii

List of Tables

4.1 A table showing which environment variables were located in what file, andat what location, at the outset of this project. . . . . . . . . . . . . . . . . 20

4.2 A table showing how every environment variable listed in Table 4.1 belongsto one of three categories, with the omission of APPLICATION_SETTÌNGS. 21

7.1 Representative values of the VM running the CMB test-server. . . . . . . 487.2 Linux command “hdparm” device & cache read averaged (and the dataset’s

variance) benchmarking results of backends used in testing. Backend de-vices without underlineare running on their eMMC Module, and the one(s)with are running on their MicroSD card. . . . . . . . . . . . . . . . . . . . 49

7.3 Key OS and Software stats of the CMB test-server. . . . . . . . . . . . . . 517.4 Key OS and Software stats of the CMB test-backends. . . . . . . . . . . . 52

8.1 Average runtime for “Hello World” submissions in each Nth set, executedwith only one backend polling the test-server at a time. . . . . . . . . . . 58

8.2 Average runtime for “Shortest Path” submissions in each Nth set, executedwith only one backend polling the test-server at a time. . . . . . . . . . . 60

8.3 Average runtime for “Hello World” and “Shortest Path” submissions ofeach Nth set, executed with all three backends polling the test-system si-multaneously, and the absolute difference between the current and previousset’s average. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.4 Average runtime for “Hello World” and “Shortest Path” submissions ofeach Nth set, executed with backends dev1 and dev2 polling the test-system simultaneously, and the absolute difference between the currentand previous set’s average. . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.5 Average (µ) and variance (σ) of the average runtime differences values fromBenchmark tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

8.6 The speedup of the average runtimes per set in tests 4 and 5, divided bydev3 ’s Benchmark tests’ average timings per set. . . . . . . . . . . . . . . 67

ix

List of Figures

2.1 CMB System Architecture. Source: Natvig et al. (2015) . . . . . . . . . . 92.2 An illustration of how ZeroMQ could be utilized. . . . . . . . . . . . . . . 112.3 An example of how CMB could utilize Celery. . . . . . . . . . . . . . . . . 12

4.1 Proposed CMB Database Schema. . . . . . . . . . . . . . . . . . . . . . . 284.2 Diagram showing the activity relationships between the actors in the pro-

posed CMB implementation. . . . . . . . . . . . . . . . . . . . . . . . . . 304.3 Diagram of Program Flow on Backend. . . . . . . . . . . . . . . . . . . . . 33

8.1 Average runtime for “Hello World” submissions in each Nth set, executedonce with each backend singly polling the test system. . . . . . . . . . . . 59

8.2 Average runtime for “Shortest Path” submissions in each Nth set, executedonce with each backend singly polling the test system. . . . . . . . . . . . 61

8.3 Average runtime for “Hello World” and “Shortest Path” submissions ineach Nth set of test 4, when executed with all three backends polling thetest-system during the test. . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.4 Average runtime for “Hello World” and “Shortest Path” submissions in eachNth set of test 4, when executed with backends dev1 and dev2 polling thetest-system during the test. . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.5 Parallelization speedup trends from tests’ averages. . . . . . . . . . . . . . 68

x

List of Listings

4.1 The CMB start-up script used for both starting and stopping CMB, at theoutset of this project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.2 The infinite while-loop of the backend process, polling the Flask web serverfor submissions to profile. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

9.1 The CMB crontab script used for monitoring the server processes of CMB,at the outset of this project. . . . . . . . . . . . . . . . . . . . . . . . . . . 72

9.2 The proposed system implementation of the automatic monitoring of back-ends. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

9.3 How the proposed system implementation handles the potential race-conditionof multiple backends polling for submissions from the web server runningwith several Gunicorn “worker” threads. . . . . . . . . . . . . . . . . . . . 75

9.4 The lines of Python code where we believe the Gunicorn bug can occur. . 77A.1 Environment variables source-script of test-server. . . . . . . . . . . . . . 93A.2 Environment variables source-script of backends. . . . . . . . . . . . . . . 94A.3 Secret/sensitive environment variables of test-server. . . . . . . . . . . . . 95A.4 Secret/sensitive environment variables of backends. . . . . . . . . . . . . . 95A.5 Machine-specific environment variables of test-server. . . . . . . . . . . . . 96A.6 Machine-specific environment variables of backends. . . . . . . . . . . . . 97A.7 Test-server’s Gunicorn start-script/config. . . . . . . . . . . . . . . . . . . 97C.1 OS and Kernel information of test-server. . . . . . . . . . . . . . . . . . . 101C.2 CPU information of test-server. . . . . . . . . . . . . . . . . . . . . . . . . 102C.3 Memory information of test-server. . . . . . . . . . . . . . . . . . . . . . . 103D.1 OS and Kernel information of backend dev1. . . . . . . . . . . . . . . . . 105D.2 OS and Kernel information of backend dev2. . . . . . . . . . . . . . . . . 106D.3 OS and Kernel information of backend dev3. . . . . . . . . . . . . . . . . 106D.4 CPU information of backend dev1. . . . . . . . . . . . . . . . . . . . . . . 106D.5 CPU information of backend dev2. . . . . . . . . . . . . . . . . . . . . . . 107D.6 CPU information of backend dev3. . . . . . . . . . . . . . . . . . . . . . . 108D.7 Memory information of backend dev1. . . . . . . . . . . . . . . . . . . . . 110D.8 Memory information of backend dev2. . . . . . . . . . . . . . . . . . . . . 111D.9 Memory information of backend dev3. . . . . . . . . . . . . . . . . . . . . 112

1

LIST OF LISTINGS

2

Chapter 1Introduction

This chapter will first explain the Motivation for this Thesis in Section 1.1, before detailingthe Project Goals of the Master Project in Section 1.2. Finally, Section 1.3 lists thestructure of this Thesis.

1.1 Motivation

With supercomputing clusters (colloquially known asHigh-Performance Computing (HPC)centres) being a viable, albeit prohibitively expensive, alternative for computationally in-tensive workloads, there is a lot of money invested in HPC. However, while initial costsfor HPC centres are often staggering, they also tend to accrue an equally staggering costin electrical bills, just for keeping the system online and running (Subramaniam and Feng,2010).

This has prompted the HPC community to search for new, more “energy efficient” solu-tions. The search for more architectures viable for HPC has spawned many efforts, suchas the Mont-Blanc (MB) EU project (Rajovic et al., 2013).

The MB project aims to design a new type of computer architecture capable of settingfuture HPC standards worldwide, built from energy efficient solutions used in embeddedand mobile devices. The MB EU project started this quest with Rajovic et al. (2013),aiming to answer the question of whether mobile, ARM-based, System(s)-on-Chip (SoC)can help reduce the cost of HPC, due to their proliferate abundance in embedded devicessuch as smartphones. The MB project base this question on the premise that the x86architecture still dominated the TOP500 list of supercomputers in the world in June 2013.

3

Chapter 1. Introduction

During the literature research for this project, it seems that hardware is the primaryplatform on which the focus of energy efficiency currently resides.

However, while there is a considerable amount of effort put into finding the next generationenergy efficient hardware, software will always remain an important “half” of energyefficient HPC. Energy efficient software is simply software which, while competitivelyachieving the same results as the “traditional” software, also consumes less energy duringexecution.

The Climbing Mont Blanc (CMB) project (Støa and Follan, 2015), based on the MBproject, attempts to aid the search for energy efficient software. Støa and Follan (2015),along with Natvig et al. (2015), built a system for measuring the energy efficiency of code,which runs on an ARM-based architecture also used in Samsung Exynos (smartphone)processors. This system permits the user to upload their code via a web-browser userinterface1, have their code run on an Odroid-XU3 development board, and have thetiming and energy consumption readings returned to them upon successful execution.

The plans for CMB, going forward (Magnussen, 2015), include:

• To handle a larger user base:– Support a higher frequency of submissions.– Support a higher concurrency of submissions.

Currently, CMB consists of one web server, which utilises a single Odroid-XU3 card asa backend, to execute all submitted code profilings, one at a time. While there exista plethora of web technologies to enable better load-balancing of web servers (such asGunicorn (2010)), we are currently not aware of a technology which would permit uswith relative ease and reliability load-balance the code profilings/executions on multipleOdroid-XU3 cards.

Technologies like Vagrant (Gajda, 2015) and Docker (Merkel, 2014) might have been ofuse. However, without extensive testing, we doubt the reliability of the energy consump-tion readings of a system implementing these technologies, due to the added complexi-ties/overhead incurred by either of these.

1.2 Project Goals

In this section, we summarise the goals we have set for this project, based upon the taskslisted in the Problem Statement, and inform the reader where in the report the detailingof their realisations are located.

1https://climb.idi.ntnu.no

4

1.3 Thesis Structure

1.2.1 Automatic System Monitoring and Recovery

With the Motivation from Section 1.1 in mind, the Problem Statement lists four subtasks,and an additional six subtasks if time permits. Of the four mandatory subtasks, only thefirst one does not mention/involve a dispatcher. Said subtask says “Study the existing so-lution in CMB for automatic system monitoring and recovery and suggest improvements”.

Currently, the system relies on the administrators with access to the server and backendto log in and manually restart any component(s) that crash/go down. Thus, a goal ofthis project is to simplify the start-up process, such that a service like Upstart (Upstart,2006) or Systemd (Poettering et al., 2010) may monitor the processes of the system, andrestart them as required.

The creation/implementation of an automatic monitorization and recovery system throughthe use of Systemd or Upstart was not accomplished in the allotted time. The reason forthis is elaborated in the Preface, while the efforts made are described in Section 4.2 anda potential solution is described in Section 9.1.

1.2.2 The Dispatcher

Subtasks 2-4 from the Problem Statement either specify the creation/addition of a “dis-patcher” to CMB, or rely on an existing one. The job of the dispatcher is to dispatchcode submissions users have uploaded to CMB, to the backends which perform the codeexecution/profiling. And hence the creation/addition of such a dispatcher in the CMBcodebase is one of the goals we have set for this project.

While Celery (Solem, 2009) at first was considered to be an apt solution for parallelizingCMB’s backend2, it was decided by the people behind CMB in week 10 of this contrac-tually allotted 21-week project that introducing Celery into CMB would introduce toomuch complexity into the system3.

Thus, the parallelization of the backends in the proposed system solution is realisedthrough extending the REST API of the server, and having the backends added pollingthis API every so often. The realisation of this feature is further detailed in Section 4.5.

1.3 Thesis Structure

With the very technical emphasis of this master project, the structure of this reportstands a little out from the perceived norm. In this report, we focus on presenting a

2 Hence, a non-negligible amount of time and effort was spent in the attempt of implementing the useof Celery in CMB, in the first half of this project. See the Preface for more information.

3 Celery and its complexities are detailed in Section 2.4.

5

Chapter 1. Introduction

robust, dependable proposed system implementation, with which we hope to fulfil asmany goals set in the Problem Statement and in Section 1.2 as possible.

With this in mind, we first introduce the CMB project, and a bit about its historyin Chapter 2, where we also introduce and describe some of the tools/implementationsconsidered during this project.

Following, we continue with Chapter 3, where we list related works we were able to find,for the CMB system, and the parallelization project, at the heart of this master project.

After that, we introduce and detail our proposed system solution/implementation inChapter 4, and describe the proposed implementation changes, and their benefits tothe CMB system.

What follows in Chapters 5 and 6, are detailed and technical install instructions, forthe proposed system implementation described in Chapter 4. The reason for not havingthese two chapters as appendixes is that we consider them to be relevant4 to much ofthe report, besides the aforementioned fact that this is a very technical report, for a verytechnical master project. (Compared to how the norm of master projects often involveproving/disproving the veracity of newer, novel ideas, the efforts of this project are basedon already proven laws of software parallelization).

We then list the methodology of our tests in Chapter 7, where we also state the hypothesesof our project, before reporting (and discussing) the results of said tests in Chapter 8.

Finally, we detail what improvements we were unable to complete, due to the time con-straints described in the Preface, in addition to our thoughts on how the CMB systemcould further be improved by any future CMB system developers in Chapter 9, before weconclude in Chapter 10.

4 And they’re referenced repeatedly throughout the report.

6

Chapter 2Background

This chapter starts with a summary of both the EU Mont-Blanc project in Section 2.1,and the Climbing Mont Blanc system in Section 2.2. A detailing of the Odroid-XU3hardware used as backend follows in Section 2.3. Section 2.4 details software alternativesconsidered for the implementation of the dispatcher introduced in Section 1.2.

2.1 Mont-Blanc, The EU Project

The Barcelona Supercomputing Center (BSC) coordinates the Mont-Blanc project (BSC,2011), which, since October 2011, has had the aim to design a new type of computerarchitecture capable of setting future global HPC standards, built from energy efficientsolutions used in embedded and mobile devices. Their long-term goal is to provide Exas-cale performance using 15 to 30 times less energy than current architectures.

In 2013, phases 1 and 2 of the MB project were given a budget of 22 million e, of which16 million e were granted by the European Commission. The time extension provided bythe final 8 million e from the European Commission in 2013, permitted BSC to extendMont-Blanc project activities until September 2016.

The third phase of the MB project (coordinated by Bull (2016), the Atos brand fortechnology products and software), started in October 2015 got funded by the EuropeanCommission under the Horizon 2020 programme. Its aim is to design a new high-endHPC platform that can deliver a new level of performance/energy ratio when executingreal applications.

7

Chapter 2. Background

2.2 The Climbing Mont Blanc System

In 2012, the Faculty of Information Technology, Mathematics and Engineering (IME) atthe Norwegian University of Science and Technology (NTNU) had the Energy EfficientComputing Systems1(EECS) Strategic Research Area projects running. Lasse Natvig(from IME, NTNU), proposed at HiPEAC3 2012, in Gothenburg Sweden, that the massesof young students and programmers could be utilised in the quest for knowledge wrt.energy efficient computing.

In the fall of 2014, Simen Støa and Torbjørn Follan began the development of ClimbingMont Blanc (CMB), under the supervision of Lasse. CMB (Støa and Follan, 2015), isa system with a web-frontend which permits a user to upload code to be executed andprofiled for time and energy consumption on an Odroid-XU3 development board. SinceJanuary 2015, each school semester at NTNU has had one or more subjects/activitiesutilising (and some relying), on the CMB system for competitions and/or homework.

CMB utilises a Python Flask (Ronacher, 2010) web server, with an added JS frontend builtwith AngularJS (Green and Seshadri, 2013), which serves web browsers the user-interfaceof https://climb.idi.ntnu.no. The Python-Flask server, running on an Ubuntu 14.04LTS Linux OS, is a REST API, which utilises an SQL database for its data, in addition toone Odroid-XU32 development board for profiling/executing uploaded code submissions.See Figure 2.1 for a graphical overview of CMB’s system/architecture.

Thus, the CMB system permits:

1. The creation of User accounts.

2. The creation of Administrator accounts which can create problems to which Userscan upload/submit code to in attempts to solve.

3. Administrator accounts to view all submissions made by Users on the system.

4. A ranking system based on the timing/energy consumption of submitted code, perproblem, available for all to see (global).

5. Users to belong to groups, and the groups may have individual ranking lists (pri-vate).

1http://www.ntnu.edu/ime/eecs2For more information, see Section 2.3.

8

2.3 Odroid-XU3

Figure 2.1: CMB System Architecture. Source: Natvig et al. (2015)

2.3 Odroid-XU3

The Odroid-XU3 (www.hardkernel.com, 2016) currently serves as the backend of the CMBsystem. (Støa and Follan, 2015) report that it has a Samsung Exynos 5 Octa (5422) chip,which has four ARM Cortex-A15 and four Cortex-A7 cores, making it a heterogeneousmulti-processing platform using ARM big.LITTLE technology. The ARM big.LITTLEtechnology reportedly enables seamless and automatic movement of workloads to appro-priate CPU cores based on performance needs.

(Støa and Follan, 2015) also report that the Odroid-XU3 has an ARM Mali-T628 GPU,which supports OpenGL ES 3.0/2.0/1.1 and OpenCL 1.1.

Additional details of the board can be found in (Støa and Follan, 2015) and (www.hardkernel.com,2016).

2.4 Concurrency Softwares Considered

This Section describes the different software frameworks/packages considered when real-ising the Project Goals(listed in Section 1.2) of this project.

9

Chapter 2. Background

The different software/frameworks were considered with the following points in mind:

1. How would it enable concurrency among the backends?2. How would/could it support differentiation between the backends wrt. to factors like

hardware architecture and/or installed/available programming languages/libraries?3. How stable (reputed stability/usage) does the concurrency technology/implemen-

tation seem to be?4. How supported does any utilised (implemented) software seem to be, by its devel-

opers? How reliable does the software’s future support appear?

As such, ZeroMQ and Celery were the only alternatives found and seriously considered,with the above points in mind.

2.4.1 ZeroMQ

ZeroMQ (ZeroMQ, 2011) is a distributed messaging framework, which allows you toimplement your own messaging infrastructure. There are many different usage exam-ples/implementations to be found, but it was quickly decided that we would rather lookfor an alternative which required less implementation effort.

The reason why a messaging infrastructure would be needed is that at the outset of thisproject, there was no two-way communication going between the CMB server and theattached backend. All interactions between the two were Bash scripts executed on theone, which in turn executed another Bash script through an SSH tunnel on the other.

If at any point during the executions of the Bash failed, crashed, or got stuck; the CMBsystem often got so unstable that it had at best to be restarted, at worst debugged, beforeit could continue to operate.

In parallel with this master project, Sindre Magnussen continues with his work on thefrontend from the fall of 2015 (Magnussen, 2015). In his master project, running concur-rently with the master project of this report, he has implemented the use of SocketIO(Rai, 2013) into his development Git branch of the CMB system. As such, any futurecombining of the efforts of his master project and this one, discussed in Section 9.4, couldperhaps capitalise on this, if needed.

ZeroMQ have several protocols which may have suited CMB (such as the Majordomoprotocol3), but again, it would require more effort than we were interested in spendingto utilise it in CMB.

The strength of ZeroMQ lies in its versatility and seems to be used widely enough (andsufficiently supported) to be a candidate for CMB. However, this versatility comes at thecost of having to implement our own messaging infrastructure.

3http://rfc.zeromq.org/spec:7.

10

2.4 Concurrency Softwares Considered

Figure 2.2: An illustration of how ZeroMQ could be utilized.

2.4.2 Celery

Celery is a Python framework offering distributed task queues. These task queues gettasks submitted to them by producer(s). Tasks submitted to queue(s) will be executedby consumer(s) listening to said queue(s). By default, a task submitted to a queue willonly be executed once by one consumer listening to said queue.

Celery was during the first ten weeks of this project assumed to represent the best courseof action for implementing the concurrency goals of this project.

Celery works by having functions assigned to the Celery framework trigger the creationof tasks to be queued and executed by the aforementioned queued and consumers, re-spectively. The way Celery uniquely identifies functions is by using the Python task(function) signatures, which are the result of the absolute import package path, and thetask’s (function’s) function definition. The Python PATH is where these signatures aredefined.

As a consequence, identical task signatures must be present in both the producer andconsumer, and the developer must be aware of any differences in the body of the functionto which the task signature corresponds. Thus, the use of Git makes it straightforward,to have multiple copies of the same code base (and thus identical task signatures) ondifferent machines.

Thus, when a producer submits a task to a queue to which one or more consumers are

11

Chapter 2. Background

listening, Celery will (with its default set-up) ensure that the task is only executed onceby one consumer. If the task is submitted to multiple queues, it will be executed once perqueue, perhaps even by the same consumer, if said consumer is listening to the relevantqueues.

If so desired, a task can also be submitted multiple times to one queue. Celery ensuresthat each tasks-submission in the queue gets uniquely identified by the relevant systems,and with default settings, each task-submission in the queue will still only be executedonce by a consumer listening to said queue.

Figure 2.3 illustrates how a CMB implementation using Celery would rely on the differentcomponents which gives Celery its complexity, e.g. RabbitMQ.

Figure 2.3: An example of how CMB could utilize Celery.

However, Celery needs a message broker (which ZeroMQ could have been implementedas) on which to create/maintain its queues and their tasks. Celery itself supports severaldifferent brokers4 and backends (backends used for the results of the tasks executed).

As the reader can see from Figure 2.3, Celery would encompass, and ensure the functioning4http://docs.celeryproject.org/en/latest/getting-started/brokers/.

12

2.4 Concurrency Softwares Considered

of, everything within the turquoise square. RabbitMQ would be the message broker,which Celery would rely upon and utilise to realise the functionality described with itsconsumers, producers, and queues.

So while ZeroMQ offers neither the framework of consumers, producers, and queues (northe transport/message layer of RabbitMQ), it instead offers the flexibility of more “open”slate.

As such, Celery gives for free that which ZeroMQ does not, automatic dispatching whichensures a submission is only executed once by a target backend (through the use ofqueues), in an already existing and documented Python framework, in use by severalindustry giants, like Opera, Google, and Facebook.

13

Chapter 2. Background

14

Chapter 3Related Work

In this chapter, we list the related work we were able to find in Section 3.2, but first wegive a bit more background on what CMB is, can (and perhaps should?) be consideredas, and what else like it there is in the world.

3.1 Online Judges - Websites for Competitive Pro-gramming

Crowdsourcing can be defined as, “the act of taking a job traditionally performed by adesignated agent (usually an employee) and outsourcing it to an undefined, generally largegroup of people in the form of an open call” (http://www.crowdsourcing.com/, 2016).

With that definition in mind, the case that CMB represents a system enabling the crowd-sourcing for more energy efficient software can be made. However, CMB is not the onlysystem or platform to which program submissions can be sent/uploaded, and evaluated.

Due to the time constraint detailed in the Preface, we were unable to complete theresearch for related work of Online Judges (OJs) to a satisfactory extent. However, oneof the previous projects on the CMB system (Magnussen, 2015) references multiple otherOnline Judges which have varying degrees of popularity. Of these, the CMB system itselfcan be said to have been (at least in part) inspired by the likes of Kattis (Kattis, 2016),which is yearly used by the International Collegiate Programming Contest (ICPC).

However, like most of the others referenced by (Magnussen, 2015), Kattis focuses on thetiming efficiency of a program submission and pay little to no regards for the energyconsumption of the submission.

15

Chapter 3. Related Work

Kattis, and many of the other OJs that are out there (including CMB) work very similarly,with a straight-forward process:

1. Create a set of problems, to which program submissions can be submitted to thesystem to solve.

2. Have a system through which teams or individuals can upload their submissions,and keep track of which submissions was uploaded by whom.

3. Have the OJ be able to measure each submission (consumption of time/energy/mem-ory/something else), and store the measurements with a relationship connecting themeasurement to its submission.

4. Additionally, most, if not all OJs include the following:

• The ability to show results on a scoreboard, publicly or otherwise available forjudges/contestants.• Deadlines within which submissions must be uploaded, so that the measure-ments of the submissions will be valid for any potential competition/score-board.

3.2 Backend Parallelization Projects

As previously stated in Section 3.1, in the time remaining for this project, we were unableto find references to other backend parallelization projects in computer science literature.

However, it is the opinion of the author of this report that there may not be that muchpublicly available out there, even for scholars looking searching through literature behindpay-walls.

The reasoning behind this is two-fold. First, it is the belief of this author that withcurrent attitudes of not allowing potential competitors, nor anyone who might representa security risk, gain insight into the workings of backends of most complicated IT systems.

Second, there seems to be little literature to be found at all, regarding technical imple-mentations of parallelization in backends of systems. (Qian, 2012) lists multiple paral-lelization tools, who without specifying the scope of parallelization tools to be evaluated,lists no backend software implementations per say, but rather discusses the automationof programming tools which are designed to write parallelized software (and the abilityof compilers to parallelize code). As stated in Chapter 1, this report is a very technicalreport, with little to no focus on any new/novel ideas and their merits.

With this second reason in mind, it seems to the author of this project that it mightbe very well plausible that there exist few, if any papers in computer science researchliterature that can be described as “related work”.

16

3.2 Backend Parallelization Projects

However, several videos of different developers at different companies lay claim on YouTube(YouTube, 2016) that they make use of Celery. In fact, three companies (Instagram,Mozilla, and AdRoil) all pride the bottom of the front-page of Celery’s home websitehttp://www.celeryproject.org/ under the heading of “Who is using celery”.

We were, unfortunately, unable to find any written, publicly available technical reportson any such efforts.

17

Chapter 3. Related Work

18

Chapter 4Proposed System Solution

In this chapter, we detail the implementation of the proposed system solution/ imple-mentation. The proposed system implementation has been developed with the intent tofulfill the Project Goals listed in Section 1.2 in a robust, and dependable manner, withwhich the CMB system can grow.

First, we describe our perceived outset state of CMB, from when this master project wasstarted. Then, we continue with describing how the proposed system implementationsupports the Project Goal of Automatic System Monitoring and Recovery. The chapterthen continues with the efforts expended in this project to upgrade the CMB code basefrom Python 2.7 to Python 3.4, to not only provide more utilities for the rest of thisproject but also to help future-proofing the CMB project.

After that, the chapter continues with describing the database changes necessitated tosupport the development of the Dispatcher in this proposed system implementation, beforedetailing the implementation of the actual Dispatcher itself, first introduced in Section 1.2.

Finally, we also detail the implementation efforts made on the code base for the Backends(cmb-board Git repository) of the CMB system.

4.1 The outset state of CMB

At the outset of this project, there were issues with the CMB system we felt had to beaddressed before our work could begin in earnest. These were:

• The random spread of where environment variables necessary for CMB’s successful

19

Chapter 4. Proposed System Solution

execution were located.• The lack of a simple and robust configuration system which could easily (and withproper oversight) permit configuration changes.

• A simplified and more robust start-up script (and process), so that automatic systemrecovery (and monitoring) could be implemented in an efficient manner.

All of the above points are related to each other, the improvement of one helps theimprovement of the others. The outset state of CMB has environment variables necessaryfor its start-up and execution located in the following locations:

Table 4.1: A table showing which environment variables were located in what file, and at whatlocation, at the outset of this project.

Location Environment Variables~/.bash_profile

• APPLICATION_SETTINGS(Basically a config file)

• CMB_MAIL_USERNAME• CMB_MAIL_PASSWORD• CMB_TOKEN_SECRET• CMB_SECRET_KEY

~/cmb/server/cmb-flask/server.cfg(This is the config file APPLICA-TION_SETTINGS points to).

• SERVER_PORT• BOARD_IP• MALI_DIR• FLASK_DIR• FRONTEND_DIR• UPLOAD_FOLDER• MAIL_SERVER• MAIL_PORT• MAIL_USE_TLS• MAIL_USE_SSL• GUNICORN_LOG_LEVEL• VERSION

(“dev” or “prod” for Production/de-velopment)

~/cmb/server/cmb-flask/crontab.txt • APPLICATION_SETTINGS

(Hardcoded to refer to the aboveserver.cfg).

~/cmb/server/cmb-flask/scripts/init_cmb.sh • logfile

(Where the CMB processes, such asGunicorn, log their output to.).

20

4.1 The outset state of CMB

Location Environment Variables~/cmb/server/cmb-flask/scripts/gunicorn_start • NAME

• USER• GROUP• NUM_WORKERS

In addition to the the five locations listed above, there are more variables which mightqualify as “environment variables” in the Python code files located in ~/cmb/server/cmb-flask/source/*.py, but we chose to leave those for another effort, another time. All ofthe environment variables in Table 4.1 can be put into one of three categories, as shownin Table 4.2.

Additionally, examples of both the config files used in the testing of the proposed systemimplementation (with their included environment variables used for the tests detailed inChapter 7), and the bash script which easily lets one source the needed environment files(as demonstrated in both Chapters 5 and 6), can be found in Appendix A.

Table 4.2: A table showing how every environment variable listed in Table 4.1 belongs to oneof three categories, with the omission of APPLICATION_SETTÌNGS.

Category Environment VariablesMachine-specific

• SERVER_PORT• FLASK_DIR• VERSION

(“dev” or “prod” for Production/de-velopment)

• USER(Name of OS user, process(es) is(are)executed with).

• GROUP(Name of OS group process(es)is(are) executed with).

21

Chapter 4. Proposed System Solution

Category Environment VariablesProcesses-specific

• GUNICORN_LOG_LEVEL• logfile/GUNICORN_LOG_FILE

(Where the CMB processes, suchas Gunicorn, log their output to.It has thus been renamed to “GU-NICORN_LOG_FILE” in this pro-posed system implementation).

• NUM_WORKERS• MAIL_USE_TLS• MAIL_USE_SSL

Secret / Sensitive• MAIL_PORT• MAIL_SERVER• CMB_MAIL_USERNAME• CMB_MAIL_PASSWORD• CMB_TOKEN_SECRET.• CMB_SECRET_KEY

4.1.1 Proposed solution to environment variables/settings

Thus, the proposed system solution has consolidated the spread of these environmentvariables/settings. In the proposed system solution, a new folder has been added to thecmb-flask (and cmb-board) directory; configs.

The proposed concept is that in this (these) folder(s), any files with the string “secret” inthe name, will be ignored by Git, and thus never added (as it never should be) to the Gitrepository/commit history. Meanwhile, machine-specific environment variables/settings,as well as process-specific ones can also reside here, and be copied/spread to new machinesthrough Git at a developer’s wish. As such, all other scripts, processes, and programs inthe CMB project will always know where to look for any setting they may need.

As an example, cmb-flask/configs may (and should) contain the equivalent of the fol-lowing:

• crontab.txt• secrets.cfg(May (should?) be named something else than just “secrets”, so as to differentiatebetween “dev”, “test”, and “prod”).

• machine-settings.cfg(May (should?) be named something else than just “machine-settings”, so as todifferentiate between “dev”, “test”, and “prod”).

22

4.2 Automatic system monitoring and recovery

• gunicorn-config.cfg(May (should?) be named something else than just “gunicorn-config”, so as todifferentiate between “dev”, “test”, and “prod”).

Having consolidated the location for all environment variables required by the CMBsystem’s processes makes the start-up not only simpler but also more robust. Also, thissystem makes it clearer where what environment variables should be located, and whatenvironment variables each file represents and should contain. As long as the directorystructure inside cmb-flask (and cmb-board) is upheld, the excerpts of the start-upscripts listed in Subsection 5.3.1 show how the new system would work, compared withthe old.

Note, for instance, that with the new system, the initiating user of the CMB-processesdoes not need to remember to source five (5) specific files (residing in differing locations)him-/her- self, nor activate the virtual environment path variables.

These efforts simplify the implementation of the automatic process monitoring and re-covery substantially, further detailed in Section 4.2.

4.2 Automatic system monitoring and recovery

With the strategy regarding environment variables laid out in Subsection 4.1.1, initiatingthe CMB processes on e.g. the server is much simplified.

An automatic monitoring and recovery implementation needs to be able to achieve twothings:

a) Monitor when the system either crashes or becomes unresponsive,

b) and start the system as needed.

As stated in Subsection 4.1.1, the commands required to initiate the start-up of the CMBprocesses(es), can be found in Sections 5.3 for the proposed server implementation, andSection 6.3 for the proposed backend implementation.

For comparison purposes, Listing 4.1 shows the start-up script used for the CMB systemat the outset of this project. This file is with the outset state of CMB only usable by onespecific user on the machine, which all developers / maintainers of the CMB system mustbe able to access.

Unfortunately, due to the time constraints detailed in the Preface, the proposed systemimplementation in this report does not include an analogous start-up script, for neither theserver nor backend, nor an equivalent of the “checkOnline.sh” crontab1 script. However,

1 Citation: help.ubuntu.com/community/ (2016).

23

Chapter 4. Proposed System Solution

Section 9.1 describes how these things may be achieved, in the future of the CMB system.

Note that with the “init_cmb.sh” file shown in Listing 4.1, environment variables such asAPPLICATION_SETTINGS must already have been set (sourced) for the script to work.(This is why, with the outset state of the CMB system, that the script can only be initiatedby a user who has the needed environment variables stored in its ~/.bash_profile ).

In contrast with the start-up script in Listing 4.1, the source commands listed in Sec-tions 5.3 and 6.3 can be input into a start-up script, completely eliminating the need tolog into a system as a particular user (which is arguably a security risk), and gives theCMB system the potential to not only initiate the start-up of CMB processes by differentusers on a system/machine, but also have a log of who initiated what, when.

Listing 4.1: The CMB start-up script used for both starting and stopping CMB, at the outsetof this project.

1 #!/bin/bash2 #FRONTEND_DIR and FLASK_DIR is defined in server.cfg3 . $APPLICATION_SETTINGS4 logfile="/srv/climber/cmb/server/cmb−flask/logs/startup.log"5 set −e6 function start_cmb {7 if screen −list | grep −q "cmb"; then8 echo "CMB␣allready␣running.␣Try␣stopping␣before␣starting"9 exit 010 fi11 # screen −d −m −S cmb12 # screen −S cmb −X stuff "cd $FRONTEND_DIR && gulp $VERSION13 #"14 cd $FRONTEND_DIR && gulp maintenance 2>&1 >> $logfile15 #sleep 216 echo "starting␣server..."17 sleep 118 # screen −S cmb −X screen $FLASK_DIR/scripts/gunicorn_start19 { $FLASK_DIR/scripts/gunicorn_start 2>&1 >> $logfile &} > /dev/null20 disown21 sleep 522 echo "starting␣push..."23 #screen −S cmb −X screen24 #sleep 125 #screen −S cmb −p 2 −X stuff "export LC_ALL=’’26 screen −d −m −S cmb27 screen −S cmb −X stuff "source␣$FLASK_DIR/../venv/bin/activate␣&&␣cd␣

$FLASK_DIR/source␣&&␣python␣$FLASK_DIR/source/push.py28 ␣␣␣␣␣␣␣␣"29 echo "starting␣frontend..."30 cd $FRONTEND_DIR && gulp $VERSION 2>&1 >> $logfile

24

4.2 Automatic system monitoring and recovery

31 echo "CMB␣started"32 }3334 function stop_cmb {35 # echo "stopping screen..."36 set +e37 pkill gunicorn38 set −e39 screen −X −S cmb quit40 sleep 141 echo "CMB␣stopped"42 }4344 case "$1" in4546 start) start_cmb47 ;;48 stop) stop_cmb49 ;;50 restart) stop_cmb51 sleep 152 start_cmb53 ;;54 ∗) echo "Please␣pass␣’start’,␣’restart’␣or␣’stop’␣as␣argument"55 esac

Finally, on the subject of automatic process monitoring, the aforementioned “checkOn-line.sh” script can be found in Listing 9.1. While simple in principle, there are a several“gotcha’s” with the Bash implementation which seems to have gone unnoticed by thedevelopers of this script.

• It is with the intent of avoiding such pitfalls commonplace with Bash code, that therewas (and still is) a strong desire at the outset of this project by all who have recentlyworked on the CMB system, to convert the Bash scripts running on both the serverand backend into Python code.

The realization of a monitoring system in the proposed system implementation of CMBis already partly implemented, with there being a REST API call implemented in the filecmb-flask/source/routes/backends.py, which can be utilized (or further modified) tohave the server report how long ago since a backend polled the server. The code forthis REST API call can also be expanded to more carefully take into consideration that abackend which hasn’t recently polled, may be busy with an assigned submission to profile.

25

Chapter 4. Proposed System Solution

4.3 Upgrade from Python 2.7 to 3.4

Python2 the programming language has had a checkered history, with regards to itsevolution.

As with any programming/scripting language, its developers want to balance the wishfor greater features implemented into the language while ensuring backward compatibil-ity. The two extremes can often be mutually exclusive for any software product (notjust programming languages), and focusing too much on one (even if it’s just “tidyingup” or improving the internals of the product/language), may nevertheless often end upneglecting users.

And with Python 3.0 being released in 2008, and the final 2.7 version released in mid-2010,the jump from Python 2 to 3 had been made with less regard for backward compatibility,due to a wish to clean up Python 2.7 properly3. Unfortunately for Python, this createdsomewhat of a split between Python 2 and Python 3, with the user base split on whichversion they wanted to use.

The Python 2 user base wanted to continue using the no-longer-receiving major updatesPython 2 due to all of the scripts, programs, and efforts spent in Python 2, and thenon-negligible cost in the effort of upgrading all existing Python 2 code to Python 3.Meanwhile, the Python 3 user base wanted to capitalize on the better Unicode support (alltext strings now being Unicode by default), saner bytes/Unicode separation, in additionto many other improvements and utile additions.

Two areas utile for CMB in the effort of replacing the bash-scripts executing code pro-filings, which got improvements in Python 3, was the “OS” library4 and “Subprocess”library5.

The Python OS library (module in Python terminology) offers tools for file manipula-tion, with greater reliability and utility than the Python 2 version does. Things such asos.makedirs() now being able to construct all non-existing leaf-folders necessary, and per-mitting os.chmod() to accept a file descriptor as input, in addition to following symlinks,and more.

While in the Python Subprocess module offers the tools for spawning new processes,obtaining their return codes, and connecting to their input/output/error pipes. All ofwhich CMB dearly needs for replacing as much as possible of the execution of the codesubmission profilings. What is new in version 3.3, (and 2.7 does not have), is the supportfor giving spawned processes a timeout limit. This has long (and often) been an issue forCMB with its implementation at the outset of this project.

2https://www.python.org/about/3https://wiki.python.org/moin/Python2orPython3.4https://docs.python.org/3.4/library/os.html.5https://docs.python.org/3.4/library/subprocess.html.

26

4.4 Database changes

During this project, great effort has been expended to upgrade not only the code basefrom Python 2 to Python 3 but also to ensure that the unit tests already written for thisproject also worked as intended in Python 3.

4.3.1 Git submodules

Thus, with the added realization that a lot of the code needed on the server, would alsobe of use to the backend (such as a shared function permitting the spawning of newprocesses with an optional timeout), the proposed implementation involved convertingthe cmb-flask/source/cmb_utils folder and its Python contents into a Git submodule.

A Git submodule is its own wholly valid Git repository, but it is also simultaneouslyacting as a “sub-directory” (hence the name “submodule”) of another Git repository.This permits us only to have to deal with one set of code, instead of having copy/paste-like duplicates between both. As earlier stated, the majority of the functionality availablein cmb-flask/source/cmb_utils/*.py’s files would contain functionality utile for boththe server and the backend(s).

An added advantage of this implementation is that if there’s a bug found, the bugfix onlyneeds to be implemented once, and can then be pulled6 into the other repositories alsousing it as a submodule.

4.4 Database changes

With the implementation of the Dispatcher7, code submissions uploaded by users can berun on any eligible backend for profiling its energy efficiency and timing. Thus, it wouldbehoove the administrators of CMB to know which profiling was run on which backend(Odroid-XU3 board). This is particularly the case if a future CMB system wishes tosupport different backend architectures, as stated in the Problem Statement.

The addition of rows in tables, or manipulation of relationships between them, can bemodified both through scripted or interactive Python code (exemplified in the cmb-flask/sources/init_db.py file), or through the admin interface, created by the Flask webserver.

In Figure 4.1, the white rectangles represent the database tables which CMB had inits implementation from the outset of this project. It is worth mentioning that whilethe database schema permits an uploaded code submission to have multiple runs, thisfunctionality is utilized by the CMB today (nor in this reports proposed system imple-mentation, though we attempted to make it easier for future developers to enable it).

6Through use of the command git pull.7Detailed in Section 4.5.

27

Chapter 4. Proposed System Solution

This is discussed further in Section 9.3.

In summary, a Problem may have anywhere from 0 to N Submissions, and each Submis-sion necessitates one (and at most one) User. Each Submission may (permitted throughthe database schema, not the server-code implementation) have 0 to N Runs, of saiduploaded code submission executed on a backend, as illustrated in Figure 2.1.

Figure 4.1: Proposed CMB Database Schema.

4.4.1 Changes necessitated by the Dispatcher

Before we continue describing Figure 4.1 and the changes it represents made in the pro-posed system implementation, we want to make clear that due to the time constraintsdetailed in the Preface; there is one discrepancy between the figure and the proposed sys-tem implementation. The relationship between the SoftwareSet table and the Backendtable does not exist. Instead, the Backend table has a “many-to-many” relationship withthe Software table. Note that the proposed system implementation takes this into con-sideration, and testing shows that it works as intended, however inelegant and undesiredthis alternative is to what’s presented in Figure 4.1.

The light blue rectangles in Figure 4.1 represent the tables added in the proposed systemimplementation of this project.

Thus, a Run now needs to associate with a Backend, while a Backend may have 0 to NRuns associated with it. Each Backend, in turn, needs to associate with (and at most) 1Architecture, but several Backends may share the same Architecture. An example of an

28

4.5 The Dispatcher

Architecture would be “Odroid-XU3”, signifying the 32-bit ARM big.LITTLE CPU andARM Mali GPU cores8.

Also, a SoftwareSet consists of 1 to N Softwares (which in turn are unique by the combina-tion of name and version), with the restriction that no two SoftwareSets can be identical.Every Backend must support a SoftwareSet, and several Backends may support the sameSoftwareSet.

Likewise, a Problem must require a SoftwareSet, and multiple Problems may require thesame SoftwareSet. With the final additional requirement of Problems needing a targetArchitecture, it is through this logic that a newly spawned Run checks whether or not aBackend querying for its next job is eligible or not.

4.4.2 Potential changes for software language support

Subtasks 8 (and to a small extent 6) in the Problem Statement ask for a proposal on howthe CMB system could be expanded so as to support for code submissions in differentprogramming languages.

The green rectangles represent database tables which have not been implemented, whichif implemented could easily facilitate support for multiple languages/libraries in CMB.This is further discussed in Section 9.3.

4.5 The Dispatcher

As stated in the motivations listed in Section 1.1, CMB currently executes all uploadedcode submissions sequentially, on one set of hardware. To enable code submissions to beprofiled/executed concurrently, the implementation of a “dispatcher”, which can dispatchsubmissions to different backends, has been requested.

At first, Celery9 was considered to be an apt tool for implementing the dispatcher. How-ever, as first mentioned in the Preface, in week 10 out of the 21 weeks of the project’sduration, it was decided in a meeting with Lasse Natvig and IDI’s IT dept. representativeArne Dag Fidjestøl that Celery introduced too much complexity10, in addition to havingto the backends communicate directly with the MySQL database.

CMB is currently switching from using a sqlite3 database located on the CMB servermachine, to a MySQL database hosted on a separate machine/server. Hence, the combi-nation of RabbitMQ and MySQL as a broker (message transport), and results backend

8Detailed in Section 2.3.9 Introduced in Subsection 2.4.2.

10 With the need for a broker such as RabbitMQ, in addition to the backends having to communicatewith something like a database for results persistence.

29

Chapter 4. Proposed System Solution

(respectively) appeared to be a good fit for this project.

With the decision of not to utilize Celery having been made half-way into the project11,it was instead decided to expand and utilize on CMB’s REST API, to realize the “dis-patching” mechanism. This decision was in part sparked by the fact that implementingthe Dispatcher through the Flask REST API, instead of a Celery implementation, leavesthe Flask web server the sole agent in the CMB system interacting with the database, asdepicted in Figure 4.2.

Figure 4.2: Diagram showing the activity relationships between the actors in the proposedCMB implementation.

Figure 4.2 shows how the different actors of the system, the users, CMB administrators,backends, database, and web server interact with one another. What the dotted linedenotes is that the user base can in principle perform HTTP Requests to the Flask web

11 At the meeting described in the Preface.

30

4.6 The Backend

server REST API, as long as the firewall/connection settings of the web server permit it.

Otherwise, all interactions with the system go through the Flask web server, with theFlask web server being the only agent (as previously stated) interacting with the Database,and each of the backends. (The backends in turn only communicating separately withthe Flask web server).

The front-end is a process currently running on the same VM/machine as the Flask webserver (though it does not have to), and it serves the HTML/JavaScript web browsercontent accessible to the user base through the climb.idi.ntnu.no URL endpoint.

Additionally, in Figure 4.2, the arrows denote which actors communicate with one another(as already detailed), with the arrowhead pointing from the active agent, and to the agentthe active agent requests information from (or updates with information).

4.6 The Backend

With the REST API implementation introduced and described in Section 4.5, each back-end needs to pro-actively poll the Flask web server’s REST API, to see whether or notthere are any queued submissions awaiting energy- and timing-execution profiling.

With the added motivation of moving away from Bash scripts with insufficient reliabilityalready stated in Section 4.2, efforts were made in this project to write the pro-activesoftware on the backend in Python code to be as stable, and the least prone to unexpectederrors, as possible.

Listing 4.2: The infinite while-loop of the backend process, polling the Flask web server forsubmissions to profile.

77 def main():78 while True: # "main" infinite loop79 status_code, polling_request = poll_request()80 while status_code != 200:81 sleep(SLEEP_PERIOD)82 status_code, polling_request = poll_request()8384 print(getCurrentTimeString() + "Parsing code submission received from server...")85 try:86 request_json_data = polling_request.json()87 except Exception as e:88 print(getCurrentTimeString() +89 "Following Exception occured during parsing of received request’s JSON

buffer:\n\n{}\n"90 "\tRe−fetching code submission from server.\n".format(e))91 continue92 print(getCurrentTimeString() + "Code submission received from server parsed:")93 print(request_json_data)

31

Chapter 4. Proposed System Solution

9495 try:96 print(getCurrentTimeString() + "Preparing backend code profiling run...")97 profile_dict = run_backend_profiling_run(request_json_data, sleep_period=

SLEEP_PERIOD)98 except Exception as e:99 print(getCurrentTimeString() +

100 "Following Exception occured during execution of backend profiling run:\n\n{}\n"

101 "\tRe−fetching code submission from server.\n".format(e))102 continue103 finally:104 cleanup(request_json_data[’data’][’run_id’])105106 # Return data107 push_results(profile_dict)

Listing 4.2 illustrates some of these efforts, through showing the code for the “main infinitewhile-loop” running on the backend. It is this code which is continuously executed whenthe steps in Section 6.3 are followed. The try/except and try/except/finally code blocksare what ensures that no matter what errors or crashes occur in the Python code itself,the process running on the backend will not end until so told by outside influence. (Futuredevelopers of the CMB system must beware this fact, because if the code produces errors,it will go into an infinite while-loop infinitely producing said error).

Figure 4.3 shows the control-flow of the code (including the code in the infinite while-loopshown in Listing 4.2) running on the backend. The green, white, yellow and red boxesrepresent actions which should be easily recognized in Listing 4.2. All the steps in theblue boxes however, are all inside the run_backend_profiling_run() function call on line97 of Listing 4.2.

The stippled lines going from each of the blue boxes, and to the red box, represent thecode written to support the abortion of the profiling run at the end of any of the stepsrepresented by a blue box.

The X value in the top right yellow box is by default 12, and the same value was used inthe tests detailed in Chapter 7.

Finally, we wish to make it known, that with the folder structure in cmb-board, the intentis that it should be easy to add implementations for new architectures/backends in foldersanalogous/parallel to the cmb-board/odroid-xu3 folder.

Thus, only the code represented by the blue boxes of Figure 4.3 need be replaced, andthat code is all located in the aforementioned cmb-board/odroid-xu3 folder.

32

4.6 The Backend

Figure 4.3: Diagram of Program Flow on Backend.

An important note regarding the backends used in the tests of Chapters 7and 8 in this report:

Due to a misunderstanding between the author of this thesis, and the supervisor of thisproject, Lasse Natvig, it was discovered that Lasse Natvig had intended for us to com-pletely re-format the backends given to run tests on. However, we had understood that wewere to take caution with the system, disturbing it the least possible, so that if somethingwere to go wrong, with either the existing system or the proposed system implementation,the backends given could more easily be reverted into use with the existing CMB system.

33

Chapter 4. Proposed System Solution

Unfortunately, this misunderstanding was uncovered too late12 for there to be sufficienttime remaining to completely re-format the given backends and re-install them as de-scribed in Chapter 6.

12 See the Preface for more details of the delays and time limitations of this project.

34

Chapter 5Server Installation Instructions

This chapter (introduced in Section 1.3) lists and describes how to install the servercomponents of the proposed system implementation of this master project (discussed inChapter 4). The chapter is written so as to work as an install manual for the server-sideservices and code for the proposed CMB implementation of this report.

(Magnussen, 2015) and (Støa and Follan, 2015) have both detailed the need for the dif-ferent tools and components required by the CMB software, and any need for any com-ponents or tools inadequately described in this report, can be found in their papers.

Any new tools or components required by this proposed system implementation will eitherbe adequately explained here in the install instructions, or the chapter detailing/discussingsaid tool or component.

Keep in mind the following when reading the instructions detailed within:

• At the outset of this project, CMB had one Virtual Machine (VM) server commu-nicating with one backend.

• At both backend and server, a user was created to execute the services of CMB.

NB: The default/standard password(s) have been changed by adding “2” at the end ofthe current password string(s) used on all VMs (servers) and Odroid-XU3 cards(backends) which implement this proposed system solution.

35

Chapter 5. Server Installation Instructions

5.1 Getting the Code

With the changes to the folder structure described1 and discussed2 in the report, werecommend that the folders for the frontend and server CMB codebases (both of whichcurrently reside on the server), be located in the same directory.

Thus, it’s recommended to make one cmb/ folder, for example either in ~/ or /Documents/,in which both the two below folders get located.

1. cd into the folder you want them to end up in, and execute the following commands:git clone git@bitbucket.org:climbingmontblanc/climbing−mont−blanc.git cmb−jsgit clone git@bitbucket.org:climbingmontblanc/cmb−flask.git cmb−flask

• Both having a website on the service hosting the cmb-js and cmb-flask Gitrepositories respectively:https://bitbucket.org/climbingmontblanc/climbing-mont-blanchttps://bitbucket.org/climbingmontblanc/cmb-flask/

2. Inside the cmb-flask3 folder, execute the following command to also git pull theneeded submodule repository4:git submodule update −−init −−recursive

At the time of writing, there are several divergent branches in the different Git reposi-tories, due to there having been several different Master Projects working on/with thesystem simultaneously, with at least two of those modifying the same codebases.

• Thus, make sure you select the correct Git branches you want, and that they arecompatible with each other. The default Git branch master should be compatiblewith all the other master branches. This can be confirmed as needed with LasseNatvig and Sindre Magnussen5.

• If you want to utilize the branches (commits) which run this report’s proposed sys-tem implementation, you can execute the following commands in the two folderswhen their installation is complete:cd cmb−flask/git checkout test−chrischa−branch 6

cd cmb−js/

1Section 4.1.2Section 9.4.3cmb-flask and cmb-js can be named anything, as long as the names of the two directories differ.

They will be referred to as cmb-flask and cmb-js in the rest of this chapter.4Discussed in Subsection 4.3.1.5lasse@idi.ntnu.no and sindrma@stud.ntnu.no, respectively6 Git commit hash: 7ed3e1d8958a98b4e5cf7d5c713f9e5636b8ef3b

36

5.2 Install Instructions and Pre-Requisites

git checkout project_structure_rewrite 7

5.2 Install Instructions and Pre-Requisites

For the set-up of the server, the following instructions list the pre-requisites and remaininginstall instructions:

a) A machine-specific <X>-secrets.cfg8 config file located incmb-flask/configs/<X>-secrets.cfg. It is recommended to copy the one usedon a previously working system, and modify it as needed on the new system.

b) A MySQL Server.

• The username, password, and database name to be used on the MySQL serverstored in CMB_MYSQL_USER,CMB_MYSQL_PASSWORD, and CMB_MYSQL_DATABASE,environment variables located in <X>-secrets.cfg, respectively.

c) A Ubuntu 14.04 (or equivalent/derivative) server-machine9, which will need thefollowing:

1. The IP’s of all the backends which will be used (minimum one) stored in acomma-separated string in the BOARD_IPs environment variable in <X>-secrets.cfg.

2. A user (normally just named “climber”), which must be able to ssh to eachand every backend without being requested for password. ssh−copy−id 10 canbe used to achieve this11.

If the install happens on a IDI VM machine, it’s recommended to edit theuser’s UID in /etc/passwd to an available number under 1000, so as to en-able the sudo passwd climber command to run without being hindered byIDI/NTNU’s Kerberos..

3. The equivalent of sudo apt−get install -ing the following:build−essential gcc−5 g++−5libmysqlclient−dev libffi−devgit python3 python3−dev python−virtualenv realpath

7 Git commit hash: 910d4f7d91c60d5e5e5283e1aa4c93d4eacf7cbb8<X> being say “prod”, “dev”, or “test3”, as discussed in Section 4.1.9These install commands have been tested on Ubuntu 14.04 LTS, 15.10, and 16.04 LTS.

10See Section 6.1 for an example of how to.11 If the Python code struggles with SSH during execution, see Appendix B.

37

Chapter 5. Server Installation Instructions

openssh−server openssh−client fail2ban unattended−upgrades

Important note: (Støa and Follan, 2015; Magnussen, 2015) both report thatthe only requirement to have security updates automatically installed is toinstall the package unattended−upgrades . In the duration of this project,it has repeatedly been noticed that the backends have stated they have “Xsecurity updates pending”, without this reported number ever diminishing.Some research into the tool revealed this12 URL, which documents that thereis additional set-up required (other than just installation through the packagemanager), for the tool to automatically install security updates on the machine.

4. And running these commands to ensure the security of the machines:sudo ufw allow 22sudo ufw allow 80sudo ufw allow 443sudo ufw enable

d) The following commands executed in cmb-js:

• Need to sudo apt−get−install :npm nodejs−legacy

• And then (in the root folder of CMB frontend):sudo npm installsudo chown −R climber:climber node_modules 13

e) The following commands executed to finish the install of cmb-flask:cd cmb−flask/virtualenv −p python3 venvvenv/bin/pip install −r requirements.txtsource scripts/<X>−source_cmb_envvars.shcd source/ && ../venv/bin/python init_db.py

The last two commands sets up the database as needed, and you can readcmb-flask/source/init_db.py for what else it does if it’s a “dev” install.

f) Finally, copy the Mali_OpenCL_SDK_v1.1.0 folder and following contents with thefollowing commands into the same directory where cmb-flask is located:cd cmb−flask/../

12 https://help.ubuntu.com/community/AutomaticSecurityUpdates13Replace “climber:climber” with whatever “<user>:<usergroup>” that’s applicable for your installa-

tion, as needed.

38

5.3 Starting the Server

mkdir Mali_OpenCL_SDK_v1.1.0

On a machine which already has CMB running successfully, enter theMali_OpenCL_SDK_v1.1.0 folder, and run the following command:scp −r common/ include docs/ lib/ \Mali_OpenCL_SDK_v1.1.0_Documentation.html platform.mk samples/ \climber@<new−machine>:Documents/cmb/Mali_OpenCL_SDK_v1.1.0/

5.3 Starting the Server

• To start the Flask web server in production mode, execute the following commands:cd cmb−flask/sourcesource ../scripts/<X>−source_cmb_envvars.sh../scripts/gunicorn_start.sh

• To start the Flask web server in develpment mode, execute the following commands:cd cmb−flask/sourcesource ../scripts/<X>−source_cmb_envvars.sh../venv/bin/python manager.py runserver −r −d −h $CMB_SERVER\−p $SERVER_PORT

• To start the frontend (and not just the REST API), execute the following com-mands:cd cmb−js/node_modules/.bin/gulp local−devnpm start

5.3.1 Start-up Script Differences

Chapter 4 details how the start-up of the CMB system functioned at the outset of thesystem and describes the proposed changes to simplify and make the process more robust.Appendix A contains the sourcing script file used for the test-system used for the testsdescribed in Chapter 7, and most of the changes suggested in Chapter 4 can be recognizedin the script file.

39

Chapter 6Backend Installation Instructions

This chapter (introduced in Section 1.3) lists and describes how to install the backendsand their constituent/ needed components of the proposed system implementation of thismaster project (discussed in Chapter 4). The chapter is written so as to work as an installmanual for the backend-side services and code for the proposed CMB implementation ofthis report.

(Magnussen, 2015) and (Støa and Follan, 2015) have both detailed the need for the dif-ferent tools and components required by the CMB software, and any need for any com-ponents or tools inadequately described in this report, can be found in their papers.

Any new tools or components required by this proposed system implementation will eitherbe adequately explained here in the install instructions, or the chapter detailing/discussingsaid tool or component.

Keep in mind the following when reading the instructions detailed within:

• At the outset of this project, CMB had one server communicating with one backend.• At both backend and server, a user was created to execute the services of CMB.

NB: The default/standard password(s) have been changed by adding “2” at the endof the current password string(s) used on all VMs (servers) and odroid-xu3 cards(backends) which implement this proposed system solution.

40

6.1 Install Instructions and Pre-Requisites

6.1 Install Instructions and Pre-Requisites

The backends used in this proposed system implementation and so far in the CMB projecthave been Odroid-XU3 cards, detailed in Section 2.3. These (and any new hardware usedas backend) will need to be able to execute bash and Python code with reliable accuracywith regards to energy and timing measurements. Thus, the CMB project has so far usedUbuntu or Ubuntu OS derivatives as the OS to run on the backends; all gathered fromhttp://www.hardkernel.com/main/main.php.

The latest install on a fresh Odroid-XU3 used Ubuntu 15.101, and was executed on anUbuntu 15.10 system following the below install instructions:

1. Download the Ubuntu OS derivative image of your choice, made available for installon the XU3 at http://www.hardkernel.com/.

2. The Odroid-XU3 board can either install boot its OS from the MicroSD card, orthe eMMC module. To install the downloaded OS image onto the eMMC module,an eMMC module reader is needed. Likewise, if the OS image is to be installedonto from the MicroSD card, a MicroSD card reader is required.

It is important to ensure that the OS image chosen and downloaded supports theARM EnergyMonitor program.

3. Execute the following Unix terminal instructions to install the OS image ontowhichever card was chosen as destination2:

The following command flushes the card chosen as destination for the OS, overwrit-ing all its data in 4M blocksizes of zeroes.sudo dd if=/dev/zero of=/dev/path/to/chosen/card bs=4M conv=fsync

The remaining commands describe the actual install of the OS image onto the des-tination card:unxz <chosen OS>.img.xzsudo dd if=<chosen OS>.img of=/dev/path/to/chosen/card bs=4M conv=fsyncsync

4. After attaching the MicroSD card or eMMC module (loaded with the OS image) tothe board, boot it and connect it to a monitor through a mini-HDMI or DisplayPortconnection. An automatic login will appear for the user “odroid”, open a terminalcommand window and execute the following commands:sudo adduser climber −−home /home/climber −−shell /bin/bash

1Acquired from http://forum.odroid.com/viewtopic.php?f=95&t=18375, last accessed February2016.

2http://odroid.com/dokuwiki/doku.php?id=en:xu3_bootmode_configuration describes how toswitch between booting from the eMMC module and MicroSD card.

41

Chapter 6. Backend Installation Instructions

sudo adduser climber sudosudo useradd −s /usr/sbin/nologin −M −N −g climber −K UID_MAX=999 worker

These two commands will create the “climber” and “worker” users, and giving“climber” sudo-powers. When prompted for a password after entering the com-mand for the “climber” user, enter the password decided upon by the CMB team.(There’ll be no password request prompt for the “worker” user, as intended). Use theabove sudo adduser climber <group> command to add climber to all the groupsof the “odroid” user is a member of, to ensure its working in the installed OS. Bothusers are needed to execute the backend profiling executions on the backend.

Additionally, the following line should be added to the file opened by the sudo visudo ,to enable the worker to execute programs compiled by the “climber” user:(...)climber ALL=(worker) NOPASSWD: ALL(...)

5. Thereafter, log in with the “climber” user, and sudo apt−get install the followingUbuntu packages:

These first two lines of package names represent the tools needed for the executionof the CMB software, and the preparation and compilation of it.build−essential gcc−5 g++−5 gfortrangit python3 python3−dev python−scipy python−virtualenv realpath

The next line of package names represent the tools used for both control, access,and safety of the backend:openssh−server openssh−client fail2ban unattended−upgrades

Finally, this last line of package names represent the libraries needed for compilingthe CMB code, in addition to the libraries needed by the CMB code during execu-tion:libblas−dev liblapack−dev libffi−dev libatlas−base−dev qt4−default libqwt−dev

6. The following commands represent the absolute minimum necessary to set-up thenecessary SSH connections for CMB to work:

First, log in with the “climber” user.Then, execute the following commands in a terminal, using all defaults when prompted(by pressing <enter>):ssh−keygen −t rsa −b 4096

42

6.1 Install Instructions and Pre-Requisites

ssh−copy−id climber@<the CMB server backend will communicate with> 34

From this point on, it is recommended to execute the remaining commands throughan SSH terminal from another machine on the NTNU network (presuming that thebackend is also located within the NTNU network).

Thus, all I/O peripherals (with the exception of Ethernet and the eMMC mod-ule/MicroSD card) can be disconnected from the Odroid-XU3 board. Whenever aboard in shutdown mode is connected to power, it will automatically boot.

7. fail2ban should work out-of-the-box, and is used to enhance the safety and securityof the backend.

8. As stated in Chapter 5, unattended−upgrades has been reported by both (Støaand Follan, 2015; Magnussen, 2015) to work without any required additional set-up, beyond what is done by the package manager with which it was installed.However, with the observation that the backends keep stating that they have “Xsecurity updates pending”, and with this number only having increased duringthis project, some research into the tool revealed this online resource: https://help.ubuntu.com/community/AutomaticSecurityUpdates. This resource ex-plicitly states that there’s additional set-up required post package manager instal-lations, for this tool to automatically install security updates. For anyone utilizingthis install guide, it is recommended to follow the recommendations of this onlineresource when installing unattended−upgrades .

ufw should already be installed, and the following commands will ensure that onlyconnections from IP addresses within the NTNU network can connect to the back-end:sudo ufw allow from 129.241.0.0/16 5

sudo ufw enable

9. Next, you need to copy over the files and folders required, which are not includedin the Git repository due to Intellectual Property (IP) reasons:

ssh onto another CMB Odroid-XU3 backend which has successfully completed itsinstall, and execute the below commands inside of its cmb−board/ folder:scp −r common/ EnergyMonitor_v3/ include/ lib/ climber@<new backend>

10. Finally, to ensure more accurate and stable energy readings, remove the Ubuntuinstall’s UI with the equivalent of the following commands:sudo service lightdm stop

3 Replace “climber” as needed with whatever user name is used on the CMB server.4 If the Python code struggles with SSH during execution, see Appendix B.5 This being the IPv4 range of which NTNU holds ownership of. Any IPv4 addresses in this range

will be an NTNU address.

43

Chapter 6. Backend Installation Instructions

sudo apt−get purge lightdm

6.2 Getting the code

The installation of the backend(s) differ little from the described proceedings in previousiterations of CMB (Støa and Follan, 2015; Magnussen, 2015). Thus, while the instructionsin this chapter will re-iterate these, it will put a heavier focus on the changes necessitatedby the proposed system implementation of this master project.

All of the following instructions will only apply to one backend, but the instructions areidentical for each additional backend added.

Differing from the server install instructions in Chapter 5, the necessary code will onlyhave to reside in one single folder. After the backend (the Odroid-XU3 board) has gottenits OS and tools installed6, execute the following steps to download and prepare thenecessary code onto and on the backend:

1. cd into the $HOME folder of the “climber” user (or equivalent) created on the back-end, and execute the following command within:git clone git@bitbucket.org:climbingmontblanc/cmb−board.git cmb−board 7

cd cmb−board/

• This Git repository has a website on the service hosting the cmb-board repos-itory at the following URL:https://bitbucket.org/climbingmontblanc/cmb-board/

NB: At the time of writing, there are several divergent branches in the different Git repos-itories, due to there having been several different Master Projects working on/withthe system simultaneously, with at least two of those modifying the same codebases.

• Thus, make sure you select the correct Git branches you want, and that they arecompatible with each other (server and backend(s)). The default Git branchmaster should be compatible with all the other master branches. This canbe confirmed as needed with Lasse Natvig and Sindre Magnussen8.

• If you want to utilize the branches (commits) which run this report’s proposedsystem implementation, you can execute the following commands in the twofolders when their installation is complete:

6Detailed in Section 6.1.7cmb-board can only be named something else if this change is reflected in the cmb-flask/source/*.py

files of the server which the backend will be used with.8lasse@idi.ntnu.no and sindrma@stud.ntnu.no, respectively

44

6.2 Getting the code

cd cmb−board/git checkout test−chrischa−branch 9

2. The remaining steps detail how to copy the needed folders and their contents intocmb-board, which could not be added to the Git repository due to Intellectual Prop-erty reasons:

On a CMB server which already has CMB running successfully, enter theMali_OpenCL_SDK_v1.1.0 folder, and run the following command:scp −r common/ include docs/ lib/ \Mali_OpenCL_SDK_v1.1.0_Documentation.html platform.mk samples/ \climber@<new backend>:~/cmb−board/

On another CMB backend which already has CMB running successfully, enter thecmb-board folder, and run the following command:scp −r EnergyMonitor_v3/ climber@<new backend>:~/cmb−board/

3. The following instructions/commands are necessary for the compilation of the CMBcode and its tools:

This first command is a binary needed for the compilation/execution of EnergyMoni-tor and the cmb-flask/Makefile all user submitted code submissions currently use:sudo ln −sf /lib/ld−linux−armhf.so.3 /lib/ld−linux.so.3 10

Thereafter, add the below line to the /etc/rc.local file, but ensure you add itabove the line saying exit 0 .chmod +r /sys/devices/10060000.tmu/temp

4. Next, follow the below instructions/commands to compile the needed tools for theCMB software’s execution:cd into cmb-board/EnergyMonitor_v3/ and execute the following commands:qmakemake *

Thereafter, cd into cmb−board/mountBlanc , and execute the following com-mands:g++ −O2 ./dropCache.cpp −o dropCachechmod 4710 ./dropCache

9 Git commit hash: aacf2e562f007280270a149bb375e85f51c6ed4510 Your mileage may vary. During some installs, symlinks have been required. These instructions will

mark the following command(s) where it requirement has occurred with a “*”. All the required binaries(targeted by the symlinks) have always been present when following the instructions detailed in thischapter. All the needed symlinks have been needed in /usr/lib.

45

Chapter 6. Backend Installation Instructions

5. Inside the cmb-board folder, execute the following command to also git pull theneeded submodule repository11:git submodule update −−init −−recursive

6. Finally, to complete the installation of the CMB software, cd into cmb-board/ andexecute the following commands:virtualenv −p python3 venv

Now, be warned that the following two commands may take quite a long time toexecute (surplus of 20 minutes). It is recommended12 to append | less to eachcommand, if the terminal seems to freeze on a weird symbol mid-install or mid-compilation of a dependency:venv/bin/pip install numpy==1.11.0venv/bin/pip install scipy==0.17.1

The above two packets are present in the below requirements.txt file, but pipstill hasn’t managed to solve scipy ’s dependency of numpy in a good fashion,thus we install those two ourselves, manually, first13.

Finally, install the rest of the Python dependencies needed by the CMB backendsoftware with the below command:venv/bin/pip install −r requirements.txt

6.3 Starting the Backend

At the outset of this project (as described in Section 4.5), there was no active agent orprocess running on the single backend the CMB system previously ran with. However,with the implementation described in Chapter 4, and with the install (and execute)commands listed in this chapter, the proposed system implementation of this report nowrequires the backends to become active agents in the CMB system. It should be relativelysimple and straight forward, to have both a logging to file of the process running in thebackground, and a simple start-up script, combining the below three commands.

• To start the indefinitely running CMB backend Python process, execute the follow-ing commands:cd cmb−board/source scripts/<X>−source_odroid_backend_variables.shvenv/bin/ipython service.py

11Discussed in Subsection 4.3.1.12If the commands are executed manually.13This is why the packages libblas−dev liblapack−dev libatlas−base−dev are necessary.

46

Chapter 7Methodology

In this chapter, we describe the hardware, software, configurations, and set-up used forthe experiments this thesis reports the results of in Chapter 8. Hypotheses will be statedthroughout the chapter, wherever writing relevant to the hypotheses resides.

First, we detail the Hardware used in Section 7.1, before we list the Software and Configu-rations used in Section 7.2. Following that, we justify the use of the problem submissionswe have used in our testing in Section 7.3.

Finally, we detail the Benchmark Tests which show how a CMB implementation withonly one backend would behave with regards to timings (sequentially) in Section 7.4, andcomplete the chapter with describing the Parallelization Tests which show the impact ofthe parallelized implementation of our proposed system in Section 7.5.

7.1 Hardware & Hardware Set-Up

As previously stated in this report, the design of CMB is based on the premise of havingone server, with N ∈ [1,→ X)1 backends polling it at intervals. Thus, for testing thesystem, a server and backends were required.

The following two Subsections (7.1.1 & 7.1.2) detail what hardware was used for the testsdescribed in this chapter.

1 Where X > 1, up to the number of backends a CMB server can handle simultaneously.This is discussed further in Subsection 9.2.2.

47

Chapter 7. Methodology

7.1.1 CMB Server

The CMB Server on which these tests have been executed is a VM supplied by TekniskGruppe (Tg) at IDI, NTNU. Tg is also the ones who have provided the previous VM andDatabase resources for the CMB project. Hence, an identical VM to the already existingones in production, and the one reported in (Støa and Follan, 2015; Natvig et al., 2015;Magnussen, 2015) was requested.

Table 7.1: Representative values of the VM running the CMB test-server.

ResourceAvailableCPUcores

CPUCoreSpeed

BogoMIPS MainMemory(RAM)

Total DiskMemory

CPUop-mode(s)

Amount/ Value

3 (three) 2000.001MHz

4000.00 2 338 692KiB

15 GiB 32-bit,64-bit

Table 7.1 lists a selection of key stats of the VM used. The remaining output data is foundin Appendix C. All the stats listed were given by the following commands, executed in aterminal on the VM:

• lscpu• blockdev

• cat /proc/meminfo• cat /proc/cpuinfo

7.1.2 Backends

The backends used in the testing are the Odroid-XU3 boards detailed in (Støa and Follan,2015) and Section 2.3.

Each card used utilizes the same hardware, with an Samsung Exynos 5422ARM big.LITTLE32-bit Processor, an ARM Mali T628 (MP6) GPU, with 2 043 084 KiB of LPDDR3 RAM.The only hardware difference between the backends used in testing is that of the threecards used, one (named “dev3”) has a faulty eMMC port. Thus, the “dev3” backend uti-lizes an inserted MicroSD memory card for “Disk Memory” (HDD memory), as opposedto “dev1” and “dev2”, which use the eMMC5.0 module.

While all eMMC modules and MicroSD cards in use by the test backends are of thesame GiB capacity, any difference in “Disk Memory” capacity between the backends ispresumed to have negligible effect on the tests, since:

48

7.1 Hardware & Hardware Set-Up

• There’s sufficient memory capacity for the OS.• There’s sufficient memory capacity for the for the CMB software itself.• There’s sufficient memory capacity for the for code of the uploaded problem sub-missions.

• There’s sufficient memory capacity for the Software tools utilized by the CMB im-plementation.

• There’s sufficient memory capacity for all of the above simultaneously.• The CMB implementation (like the one proposed in this report), saves no persistentdata on backends as a result of executing a profiling run.(For accuracy and completeness, we also remind the reader that there are currentlyno logs saved on the backend, beyond what the OS and tools offer by default. Nev-ertheless, any logs can be very easily implemented, for example in combination withthe Linux terminal program screen .)

However, size limitations aside, what’s more relevant are the read/write speed differ-ences between the eMMC cards and the MicroSD card. Table 7.2 lists the averaged readspeed values collected by running the Linux memory device program hdparm 2 with thefollowing command:

sudo hdparm −Tt /dev/mmcblk0

Table 7.2: Linux command “hdparm” device & cache read averaged (and the dataset’s variance)benchmarking results of backends used in testing. Backend devices without underline are runningon their eMMC Module, and the one(s) with are running on their MicroSD card.

Backend dev1 dev2 dev3Avg. cached readspeed

1010.303MB/sec

999.823MB/sec

964.761MB/sec

Avg. cached MB/svariance (

√σ

2)5.430 6.077 7.786

Avg. buffered diskread speed

101.570MB/sec

149.765MB/sec

18.045MB/sec

Avg. buffered diskMB/s variance (

√σ

2)0.166 9.964 0.010

The average was found by running the command in an ssh terminal manually 15 timesin a row on each device, and finding the arithmetic mean of the results of the last tenexecutions, with no more nor fewer processes running on the device than when the backendruns the CMB test-system. Table 7.2 also lists the variance (σ) of the dataset used forthe arithmetic average, to give an idea of the stability of the results. dev3 denotes thebackend which utilizes a MicroSD card as its storage device, while both dev1 and dev2utilize their eMMC5.0 Modules as their storage device.

CMB has no limitations (beyond what the installed OS demands) on the memory avail-able to the programs executed on the backends during profiling. What we surmise from

2Version 9.43.

49

Chapter 7. Methodology

Table 7.2, is that while the buffered disk reading speeds differs most between the backendsrunning on an eMMC Module, and the one running on a MicroSD card, the differencesin cache read speeds are comparatively much smaller.

As long as the combined running of the test submission programs and the CMB systemson the backend don’t surpass the main memory capacity of the backend, we can safelyconclude that there’s no paging to the disks on any device. The baseline memory usageof the backends, without CMB software processes running, is approximately between 300to 400 MiB, reported by the Linux command terminal program top . We note that400 + 627 ' 1030 MiB, and 1030 MiB < 2, 043, 084 KiB. Hence, this gives our firsthypothesis:

Hypoth. I That the combined base memory requirement of the backend, and the memory re-quirement of the test submission programs executed by the CMB software processes,do not exceed the main memory (RAM) capacity of the Odroid-XU3 backends.(Thus, no paging to eMMC Module or MicroSD Card required).

(Fu et al., 2015) state that the eMMC5.0 HS400 module can theoretically achieve a maxdata transfer of 400 MB/s, which seems congruent with the results measured in Table 7.2.They also write that “The design goal of this system is to achieve the read/write speed ofeMMC array as 400/200 MB/s.” However, evidence for this claim is not found on eitherof the three Odroid-XU3 boards.

Meanwhile, (www.sdcard.org, 2016) write that MicroSD UHS-1 cards have a bus speedof 50 to 104 MB/s, depending on whether it’s an “SDR50” or “SDR104” implementation.(www.hardkernel.com, 2016) does not specify which of the two implementations comewith the Odroid-XU3 board, and we were unable to find another source detailing this.Nevertheless, a 50 to 104 theoretical max speed is congruent with the other results listedin Table 7.2.

We claim this due to the measured speeds reported in Table 7.2 fractional differences fromthe aforementioned theoretical max speeds vary only

√σ

2 = 10.111 percentage pointsfrom one-another, if we compare with the “SDR104” implementation. If we comparewith the “SDR50” implementation for the MicroSD Card, the standard deviation is only√σ

2 = 6.601 percentage points.

7.2 Software & Configurations

The CMB Server and Backends both utilize (and rely upon), a Debian/Ubuntu OS toexecute. (Støa and Follan, 2015) implemented the first iteration of the CMB system withUbuntu 14.04 on all utilized machines, and in this report, we have implemented the serverwith Ubuntu 16.04, and a new backend with Ubuntu 15.10.

50

7.2 Software & Configurations

Due to the time constraints detailed in the Preface, we were unable to get the time toattempt a backend install using Ubuntu 16.04. Thus, while both CMB backends andservers use a similar (if not same) OS, the tools, and configurations of both server(s) andbackend(s) differ to some extent. In the two following subsections, we detail where moreinformation about the software (OS, tools, and CMB) and configuration of the softwarecan be found, so as to accurately reproduce the circumstances of our tests.

7.2.1 CMB Server

Table 7.3 lists a few key stats of the server with which the tests, whose results arereported in Chapter 8, were executed. The software installed/required to run the serverimplementation of CMB proposed in this report is listed in the install instructions detailedin Chapter 5, and the full output of the commands which gave the data for Table 7.3 canbe found in Appendix C.

Table 7.3: Key OS and Software stats of the CMB test-server.

Software OS Version Kernel Version Default C/C++Compiler

CMB test-server

Ubuntu 16.04LTS

Linux 4.4.0-22-genericx86_64 GNU/Linux

gcc (Ubuntu 5.3.1-14ubuntu2.1) 5.3.120160413

Beyond the software, the only differences from a new server created with the instructionsin Chapter 5, are the configuration settings found in the:

1. Environment variables in file cmb-flask/configs/test-server.cfg,2. and the environment variables in file cmb-flask/configs/test-secrets.cfg.

• Censored versions of the two above-mentioned files can be found in Appendix A.

7.2.2 Backends

Table 7.4 lists some key stats of the backends with which the tests, whose results arereported in Chapter 8, were executed. The software required to run the backend imple-mentation of CMB proposed in this report is listed in the install instructions in Chapter 6,and the full output of the commands which gave the data for Table 7.4 can be found inAppendix D.

51

Chapter 7. Methodology

Table 7.4: Key OS and Software stats of the CMB test-backends.

Software OS Version Kernel Version Default C/C++Compiler

dev1 Ubuntu 14.04.4LTS

Linux 3.10.54+ #1SMP PREEMPTarmv7l armv7l armv7lGNU/Linux

gcc-4.9.real (Ubun-tu/Linaro 4.9.3-8ubuntu2 14.04)4.9.3

dev2 Ubuntu 14.04.4LTS

Linux 3.10.69 #1SMP PREEMPTarmv7l armv7l armv7lGNU/Linux

gcc-4.8.real (Ubun-tu/Linaro 4.8.5-2ubuntu1 14.04.1)4.8.5

dev3 Ubuntu 15.10Linux 3.10.96-78 #1SMP PREEMPTarmv7l armv7l armv7lGNU/Linux

gcc (Ubuntu/Linaro4.9.3-5ubuntu1) 4.9.3

Beyond the software, the only differences from a new backend created with the instructionsin Chapter 6, are the configuration settings found in the:

1. Environment variables in file cmb-board/configs/odroid-xu3.cfg,2. and the environment variables in file cmb-board/configs/test-secrets.cfg.

• Censored versions of the two above-mentioned files can be found in Appendix A.

7.3 Upload- and Profiling- test-problems

To test the proposed system implementation, we use the already existing “Hello World”and “Shortest Path” problems in the existing CMB system. Not only will this choicespare us the time and effort of creating new ones, but also saves us from having to ensureboth their stability and reproducibility. This effort has already been spent on “ShortestPath”, as it has also been used as a benchmark in (Støa and Follan, 2015).

Therefore we feel confident in the assertion that “Hello World” will represent somethingclose to a minimum-baseline-resources-needed problem submission for the test-system,while “Shortest Path” can be used to get more realistic results from both measuring thesystem performance during execution/testing, and energy profiling results.

Additionally, all tests whose results are listed in Chapter 8 are executed after the CMBsystem has already run tests (without stopping its processes in-between), but always rightafter having reset the CMB system (again without stopping its processes).

52

7.4 Benchmark Tests

In other words, for as long as the backends and the problem used between tests (describedin Sections 7.4 and 7.5) remain the same, said tests will not have the database reset, norhave all files previously uploaded to the test-server deleted.

If either the backends in use are changed, or a switch from “Hello World” to “ShortestPath” or vice versa occurs, the database is reset, and all files belonging to previoussubmissions are deleted on the server3.

The CMB test-server will for each and every test always be started through the use ofthe cmb-flask/scripts/gunicorn_start.sh bash script4.

Each backend, when running, polls the server immediately after start, and immediatelyafter completing a submission profiling. If the backend receives a “No more submissionsto profile at this time” response from the test-server, it waits 12 seconds before pollinganew.

7.4 Benchmark Tests

To test the implemented Dispatcher5, we need to have a benchmark of the sequentialperformance of the proposed system implementation, with which to compare. We getthis benchmark by having only one backend running (polling the CMB test-server foruploads to profile) at a time, and executing 78 uploads and profilings of each of the“Hello World” and “Shortest Path” problem test-submissions on said backend.

Hypoth. II The addition of N extra identical submissions to a set, submitted to the CMB systemto process simultaneously with the rest, should increase the amount of time it takesthe CMB system to complete profile all the submissions simultaneously submittedlinearly per submission added. This should hold true as long as there’s only onebackend polling the system, and it’s the only doing so from the submissions areuploaded, until the system is done with them.

Each of these 78 uploads and profilings were divided into twelve sets, where the Nth set(N ∈ [1, 12]) has N “concurrent” uploads and profilings. Thus, the first set has 1 uploadand profiling which is executed on the CMB system, before the second set starts its 2uploads and profilings, and so on, up until the twelfth set, which has 12 uploads andprofilings executed on the CMB system.

All uploads and profilings of the previous set NX−1 are completed (reported as failedor successful by the CMB system), before the test continues with set NX . The purposeof this wait is to time the amount of time it takes for the CMB system to handle the

3 Remember that no files/data of submissions profiled are stored on backends.4 With the exception detailed in Subsection 7.5.1.5 Detailed in Section 4.5.

53

Chapter 7. Methodology

concurrent number of uploads/submissions of a set. This will help us get an idea of theuser-experienced wait-time as the system gets more than one submission to profile at once(and thus testing the capacity and capabilities of the dispatcher).

Thus, we propose three tests, one for each of the backends;

Test 1: 2×N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path” executedsequentially on backend dev1, with each set’s submissions submitted concurrently tothe test system.

Test 2: 2×N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path” executedsequentially on backend dev2, with each set’s submissions submitted concurrently tothe test system.

Test 3: 2×N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path” executedsequentially on backend dev3, with each set’s submissions submitted concurrently tothe test system.

7.4.1 Benchmark Tests set-up

We list the average time it takes for a submission of “Hello World” and a submission of“Shortest Path” to execute from a user-experience perspective in Section 8.1, per backend.

Additionally, if time permits, we will also attempt to find and show the timing of thedifferent steps a uploaded submission goes through when uploaded to and executed in theCMB implementation.

7.4.2 Challenge due to timing difference between backends

To test the Dispatcher of the proposed system implementation, we use the given threeOdroid-XU3 cards as backends, two of which already had an OS installed and had beenemployed by the CMB project, in addition to an entirely new one (without OS installed)6.These backends have been referred to as dev1, dev2, and dev3 respectively, in this chapteralready.

Due to the differences between the backends described in Section 4.6, throughout ourexperimentation, development, and testing with the three backends, we have discoveredtiming differences between all three, even though hardware wise only dev3 stands out fromdev1 and dev2 in its use of a MicroSD card instead of the eMMC5.0 Module. Therefore,we have run the “1 − 12” benchmarks with both “Hello World” and “Shortest Path” oneach backend, with only said backend in use by the CMB test-system.

6 As explained in Section 4.6.

54

7.5 Parallelization Tests

7.5 Parallelization Tests

For testing the concurrency capabilities and capacity of the Dispatcher7 of the proposedsystem implementation, we intend to execute the 2x78 “Hello World” and “Shortest Path”test submission sets (upload and profile) with combinations of all three dev1, dev2, anddev3 backends simultaneously polling the CMB test-server for new submissions to profile.

Since we have three backends available for the tests of this project, testing with only onebackend (as proposed in tests 1, 2, 3 in Section 7.4), in addition to two, and then finallythree backends concurrently polling the CMB test-system during the execution of thetests, gives us an idea of how the system performance behaves when adding additionalbackends.

7.5.1 Parallelization Tests set-up

Thus, we propose the following tests, to not only test the capabilities of the proposedsystem Dispatcher but also to see how the system performance changes with differingamounts ([1, 3]) of backends polling the system for submissions to profile:

Test 4: 2 × N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path”,with each set’s submissions submitted concurrently to the test system, and all setsexecuted sequentially on the test-system with all three backends polling it.

Test 5: 2 × N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path”,with each set’s submissions submitted concurrently to the test system, and all setsexecuted sequentially on the test-system with backends dev1 and dev2 polling it.

Both of tests 4 and 5 will tell whether or not the dispatcher manages to parallelize theexecution of the submission’s profilings. Our intent with test 5 is to bridge the gapbetween 1 and 3 backends polling the system during test execution. Thus, the resultsof test 5 can give an idea as to the trend of the system’s behavior, when an additionalbackend is added. We chose backends dev1 and dev2 to be the backends used in test5, due to the arithmetically averaged differences between the average runtimes per set(Avg(SetN )) between dev1 and dev2 being smaller than the difference between either ofthese and dev3 .

With these tests in mind, we want to state our main hypothesis of this project, which wehave regarding the capabilities of the Dispatcher:

Hypoth. III The Dispatcher will have the effect that for N simultaneous, identical submissions(of the same problem) submitted to the CMB test-system, with M backends pollingthe server for submissions to execute, and with;

7 Detailed in Section 4.5.

55

Chapter 7. Methodology

a) 1 ≤ N ≤M , the total time the CMB system requires from the first upload untillast submission’s profiling is reported as completed, will at the most approachthe upper bound duration of 1 (one) analogous submission of said problem.

b) 1 ≤ M ≤ N and N mod M = 0, the total time the CMB system requiresfrom the first upload until last submission’s profiling is reported as completed,will at the most approach the upper bound duration of N

M analogous seriallyexecuted submissions of said problem.

c) 1 ≤M < N , the total time the CMB system requires from the first upload untillast submission’s profiling is reported as completed, will at the most approachthe upper bound duration of dNM e analogous serially executed submissions ofsaid problem.

However, due to the bug discussed in Subsection 9.2.3, we run the web-server in the“Shortest Path” halves of tests 4 and 5 with the../venv/bin/python manager.py runserver −h $CMB_SERVER −p $SERVER_PORTcommand, instead of the cmb-flask/scripts/gunicorn_start.sh bash script.

56

Chapter 8Results

In this chapter, we list the results of the proposed tests described in Chapter 7 and wediscuss the results in Section 8.3.

First, we report on the results of the Benchmark tests, described in Section 7.4 in Sec-tion 8.1, and follow it up with the likewise results from the Parallelization tests, introducedin Section 7.5 in Section 8.2. Finally in Section 8.3 we discuss the results listed Sections 8.1and 8.2, in addition to the hypotheses stated in Chapter 7, and how they measure up tothe results listed.

All reported averages in this chapter are arithmetic averages, and we justify the use of2 (two) significant decimals on the timing measurements due to the tests presenting theuser-experienced wait-time from start of first submission upload until the test-system’scompletion of the final one.

8.1 Benchmark Tests

Section 7.4 specifies tests 1, 2 and 3. These three propose the execution of “Hello World”and “Shortest Path” tests with the N ∈ [1, 12] sets set-up, executed three times, oncewith each backend singly polling the test-server.

Test 1: 2×N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path” executedsequentially on backend dev1, with each set’s submissions submitted concurrently tothe test system.

Test 2: 2×N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path” executedsequentially on backend dev2, with each set’s submissions submitted concurrently to

57

Chapter 8. Results

the test system.Test 3: 2×N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path” executed

sequentially on backend dev3, with each set’s submissions submitted concurrently tothe test system.

Table 8.1: Average runtime for “Hello World” submissions in each Nth set, executed with onlyone backend polling the test-server at a time.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 39.51 -2 52.65 13.143 63.80 11.154 76.62 12.825 88.58 11.966 101.30 12.727 114.10 12.808 126.10 12.009 138.30 12.20

10 150.40 12.1011 165.60 15.2012 177.00 11.40

(a) Average runtimes for submissions perset in seconds, executed on dev1, and theabsolute difference between the current andprevious set’s average.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 39.94 -2 51.85 11.913 64.17 12.324 76.35 12.185 88.26 11.916 100.80 12.547 113.50 12.708 125.60 12.109 137.70 12.1010 151.30 12.6011 164.20 12.9012 177.90 12.70

(b) Average runtimes for submissions perset in seconds, executed on dev1, and theabsolute difference between the current andprevious set’s average.

Nth set Avg. |Avg(Nx)−Avg(Nx−1)|1 39.61 -2 52.00 12.393 63.93 11.934 76.52 12.595 89.48 12.966 100.40 10.927 113.70 13.308 125.50 11.809 140.00 14.50

10 150.10 10.1011 165.30 15.2012 177.70 12.40

(c) Average runtimes for submissions perset in seconds, executed on dev3 , and theabsolute difference between the current andprevious set’s average.

58

8.1 Benchmark Tests

1 2 3 4 5 6 7 8 9 10 11 12Set number N

35455565758595

105115125135145155165175185

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

dsAvg. runtime per set of Hello World submissions.

(a) dev1

1 2 3 4 5 6 7 8 9 10 11 12Set number N

35455565758595

105115125135145155165175185

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Hello World submissions.

(b) dev2

1 2 3 4 5 6 7 8 9 10 11 12Set number N

35455565758595

105115125135145155165175185

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Hello World submissions.

(c) dev3

Figure 8.1: Average runtime for “Hello World” submissions in each Nth set, executed oncewith each backend singly polling the test system.

First, in Table 8.1 we list the average runtimes of each set of the “Hello World” testsexecuted on each backend. These tables also include the absolute time increments betweensets Nx and Nx−1 of the arithmetic average runtimes of all submissions in each set.

Figure 8.1’s Subfigures 8.1a, 8.1b, and 8.1c show the graphs corresponding to the growthof each set’s average submission runtime, with the same numbers as the middle columnof Subtables 8.1a, 8.1b, and 8.1c, respectively.

Following, Table 8.2 shows the same number from similar tests as 8.1, except the resultslisted in Table 8.2 stem from the tests having been executed with the “Shortest Path”problem submission.

Figure 8.2’s Subfigures 8.2a, 8.2b, and 8.2c show the graphs corresponding to the growth

59

Chapter 8. Results

of each set’s average submission runtime, with the same numbers as the middle columnof Subtables 8.2a, 8.2b, 8.2c, respectively, analogous to Figure 8.1 and Table 8.1.

Table 8.2: Average runtime for “Shortest Path” submissions in each Nth set, executed withonly one backend polling the test-server at a time.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 95.18 -2 135.40 40.223 174.20 38.804 214.70 40.505 252.00 37.306 293.30 41.307 335.10 41.808 375.20 40.109 411.60 36.40

10 452.20 40.6011 499.10 46.9012 540.30 41.20

(a) Average runtimes for submissions perset in seconds, executed on dev1, and theabsolute difference between the current andprevious set’s average.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 97.00 -2 137.90 40.903 178.70 40.804 221.60 42.905 260.20 38.606 300.20 40.007 344.90 44.708 383.80 38.909 426.70 42.9010 467.00 40.3011 505.80 38.8012 550.40 44.60

(b) Average runtimes for submissions perset in seconds, executed on dev2, and theabsolute difference between the current andprevious set’s average.

Nth set Avg. |Avg(Nx)−Avg(Nx−1)|1 110.30 -2 156.30 46.003 206.00 49.704 255.50 49.505 299.80 44.306 349.50 49.707 395.70 46.208 442.20 46.509 490.30 48.10

10 540.50 50.2011 589.10 48.6012 625.40 36.30

(c) Average runtimes for submissions perset in seconds, executed on dev3 , and theabsolute difference between the current andprevious set’s average.

60

8.2 Parallelization Tests

1 2 3 4 5 6 7 8 9 10 11 12Set number N

75125175225275325375425475525575625675

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

dsAvg. runtime per set of Shortest Path submissions.

(a) dev1

1 2 3 4 5 6 7 8 9 10 11 12Set number N

75125175225275325375425475525575625675

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Shortest Path submissions.

(b) dev2

1 2 3 4 5 6 7 8 9 10 11 12Set number N

75125175225275325375425475525575625675

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Shortest Path submissions.

(c) dev3

Figure 8.2: Average runtime for “Shortest Path” submissions in each Nth set, executed oncewith each backend singly polling the test system.

8.2 Parallelization Tests

In Subsection 7.5.1, tests 4 and 5 are specified. Like tests 1-3, they propose the executionof “Hello World” and “Shortest Path” with the N ∈ [1, 12] sets set-up, with three andtwo backends concurrently in use by the CMB test-system, respectively.

Test 4: 2 × N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path”,with each set’s submissions submitted concurrently to the test system, and all setsexecuted sequentially on the test-system with all three backends polling it.

Test 5: 2 × N sets, with N ∈ [1, 12] of first “Hello World”, and then “Shortest Path”,with each set’s submissions submitted concurrently to the test system, and all setsexecuted sequentially on the test-system with backends dev1 and dev2 polling it.

61

Chapter 8. Results

Table 8.3: Average runtime for “Hello World” and “Shortest Path” submissions of each Nthset, executed with all three backends polling the test-system simultaneously, and the absolutedifference between the current and previous set’s average.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 39.32 -2 32.36 6.963 27.34 5.024 28.85 1.515 27.02 1.836 32.10 5.087 32.04 0.068 33.89 1.859 36.23 2.3410 34.38 1.8511 37.45 3.0712 41.30 3.85

(a) “Hello World” test results.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 91.21 -2 107.00 15.793 119.00 12.004 149.00 30.005 179.10 30.106 196.20 17.107 210.60 14.408 237.80 27.209 249.00 11.20

10 267.60 18.6011 294.10 26.5012 317.50 23.40

(b) “Shortest Path” test results.

1 2 3 4 5 6 7 8 9 10 11 12Set number N

2627282930313233343536373839404142

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Hello World submissions.

(a) “Hello World” test results.

1 2 3 4 5 6 7 8 9 10 11 12Set number N

7595

115135155175195215235255275295315335

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Shortest Path submissions.

(b) “Shortest Path” test results.

Figure 8.3: Average runtime for “Hello World” and “Shortest Path” submissions in each Nthset of test 4, when executed with all three backends polling the test-system during the test.

Figure 8.3’s Subfigure 8.3b shows how the averaged user-experienced run-time for allsubmissions in a “Shortest Path” set increase with the amount of submissions in eachset, and Subfigure 8.3a shows the same for “Hello World”, with all three backends pollingthe system during the tests’ execution. Together, Subfigures 8.3a and 8.3b represent theresults of Test 4, listed in Table 8.3’s Subtables 8.3a and 8.3b.

62

8.2 Parallelization Tests

Table 8.4: Average runtime for “Hello World” and “Shortest Path” submissions of each Nth set,executed with backends dev1 and dev2 polling the test-system simultaneously, and the absolutedifference between the current and previous set’s average.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 26.06 -2 25.51 0.553 26.40 0.894 29.05 2.655 28.52 0.536 30.60 2.087 33.94 3.348 38.50 4.569 36.60 1.9010 49.45 12.8511 44.58 4.8712 43.48 1.10

(a) “Hello World” test results.

Nth set Avg. |Avg(Nx) −Avg(Nx−1)|

1 91.80 -2 101.40 9.603 134.90 33.504 160.80 25.905 187.40 13.106 200.50 32.107 232.60 25.408 258.00 27.209 280.80 22.80

10 309.00 28.2011 330.80 21.8012 346.30 15.50

(b) “Shortest Path” test results.

1 2 3 4 5 6 7 8 9 10 11 12Set number N

2426283032343638404244464850

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Hello World submissions.

(a) “Hello World” test results.

1 2 3 4 5 6 7 8 9 10 11 12Set number N

7090

110130150170190210230250270290310330350

Avg.

run

time

for

subm

issi

ons

in s

et,

in s

econ

ds

Avg. runtime per set of Shortest Path submissions.

(b) “Shortest Path” test results.

Figure 8.4: Average runtime for “Hello World” and “Shortest Path” submissions in each Nthset of test 4, when executed with backends dev1 and dev2 polling the test-system during thetest.

Figure 8.4’s Subfigure 8.4a shows how the averaged user-experienced run-time for allsubmissions in a “Hello World” set increase with the amount of submissions in eachset, and Subfigure 8.4b shows the same for “Shortest Path”, with backends dev1 anddev2 polling the system during the tests’ execution. Together, Subfigures 8.4a and 8.4brepresent the results of Test 5, listed in Table 8.4’s Subtables 8.4a and 8.4b.

63

Chapter 8. Results

8.3 Discussion

In this section we discuss the results of Sections 8.1 and 8.2, and we evaluate these resultsagainst the hypotheses introduced in Chapter 7.

8.3.1 Benchmark Tests

The graphs of Figure 8.1 and Figure 8.2 both show similar trends as the amount ofsubmissions in a set increases linearly.

Nonetheless, we are unable to find consistency in the sets’ averaged differences listed inTable 8.1 and Table 8.2. While the variance in-between these differences may be explainedby the uncertainties of networks (even though all backends and VMs/servers are withinIDI, NTNU’s LANs), we hesitate to put all the responsibility for the variance on thenetwork connections.

The backends wait 12 seconds before polling the server anew for a new submission toprofile if the previous poll did not result in any submission from the server, as stated inSection 7.3. However, this should only affect the first three submissions of any set N,N ≥3, and we can see discrepancies of up to (|Avg(10) − Avg(9)|) = 10.10, vs (|Avg(11) −Avg(10)|) = 15.20 in the serial “Hello World” test, and (|Avg(12)−Avg(11)|) = 36.3, vs(|Avg(10)−Avg(9)|) = 50.20 in the serial “Shortest Path” test on dev3 .

While dev3 differs from dev1 and dev2 with both the OS it runs with, and the hardwaredevice it uses as its HDD (detailed in Section 4.6 and Section 7.1 respectively), backendsdev1 and dev2 also have analogous discrepancies in their variance between the averagedruntime of submissions per set.

(Støa and Follan, 2015) report very stable runtime of the “Shortest Path” executionsof the uploaded C/C++ code on the backend, and that matches our observations duringtests with the proposed system implementation. This causes us to suspect the CMBsoftware implementation itself for the variances reported in the timing results. This isfurther discussed in Section 9.4.

8.3.2 Serial Hypotheses

Hypoth. I That the combined base memory requirement of the backend, and the memory re-quirement of the test submission programs executed by the CMB software processes,do not exceed the main memory (RAM) capacity of the Odroid-XU3 backends.(Thus, no paging to eMMC Module or MicroSD Card required).

Hypothesis I proposes that there should be no part of the CMB system on the backendsduring the profiling of test submissions, which require more memory than the capacity

64

8.3 Discussion

of the main memory of the backends, such that they have to page memory to theirHDD-devices. (And thus eliminating the need to account for the MicroSD vs. eMMC5.0hardware differences).

Executing a “Shortest Path” problem submission to the CMB test-system, with onlyone backend polling the CMB test-server, shows through the execution of the CMBsoftware on the backend with the /usr/bin/time −v <execute−tests−bash−command>Linux/Ubuntu terminal command, that its maximum (peak) “resident size” (memoryusage) on the backends is ' 627 MiB.

Hence, if there’s at minimum of 630 MiB available main memory to each backend when theCMB backend software is not running, that should indicate no need to page memory to abackend’s HDD-device. After completing a 78 submissions test-round of “Shortest Path”,each backend reports1 225, 229, and 239 MiB out of 1, 990/1, 990/1, 995 MiB availablemain memory (RAM) being actively used by the OS and other remaining processes onthe backends, respectively.

Thus, we consider this as supporting evidence for Hypothesis I, that the MicroSD cardshould not be what makes backend dev3 consistently perform slower than dev1 and dev2(unless the OS for some reason decides to take advantage of the MicroSD card when thereshould be sufficient RAM available). Instead, we suspect the reason to lie in the Kerneland OS differences, or the packages installed on the different systems, listed in Table 7.4and Appendix D respectively.

Hypoth. II The addition of N extra identical submissions to a set, submitted to the CMB systemto process simultaneously with the rest, should increase the amount of time it takesthe CMB system to complete profile all the submissions simultaneously submittedlinearly per submission added. This should hold true as long as there’s only onebackend polling the system, and it’s the only doing so from the submissions areuploaded, until the system is done with them.

Hypothesis II proposes that for a CMB system, with the proposed system implementationof this report that has only one backend polling it, for any set of simultaneously identicalsubmissions submitted to it, it should only be a linear increment of time for the systemto complete the set, for any additional identical submission added to the set.

We consider the graphs in Figure 8.1 and Figure 8.2 to support Hypothesis II, with whatappears to be very linear developments for between each set. Additionally, accounting forthe noise in the results listed in Table 8.1 and Table 8.2, the values in the third columnsof each subtable show

1 In the Linux/Ubuntu terminal program htop .

65

Chapter 8. Results

Backend “Hello World” σ “Hello World” µ “Shortest Path” σ “Shortest Path” µdev1 1.176 12.499 7.396 40.465dev2 0.119 12.360 4.990 41.218dev3 2.121 12.554 15.818 46.827

Table 8.5: Average (µ) and variance (σ) of the average runtime differences values from Bench-mark tests.

Table 8.5 takes values from the third columns of the subtables in Tables 8.1 and 8.2, andlists the arithmetic mean (µ) and variance (σ) of these numbers, and its numbers showus a few things:

1. dev1 requires the smallest average time increment when adding an additional sub-mission to a set executed with only this backend polling the test-server.

2. dev2 has the lowest variance in the averaged runtime values of the sets executedwith only this backend polling the test-server.

3. dev3 has both the worst variance and arithmetic mean of the two values mentionedabove.

We feel it’s important to stress that our results are from just executing each test onceand that any indications of trends between the cards should be taken with a bit of salt,especially when it comes to the variances of the tests on each backend.

However, we nonetheless feel that the data supports Hypothesis II, especially when thegreatest difference max(σµ ) of the same test and backend in Table 8.5 is ' 0.33, which isnotable, but not high enough to change the trends shown that easily.

8.3.3 Parallel Tests

The results in Section 8.2 clearly show that the proposed system implementation offersbenefits to the timing of multiple problem submissions submitted to the CMB system.

While there is a lot of noise in the results (particularly the results of the “Hello World” testsubmissions), Table 8.6 clearly show us that there are benefits to be gained by addingan additional backend to the system when processing multiple concurrently submittedsubmissions.

Though the numbers in Subtable 8.6a vary to a larger extent than those of Subtable 8.6b,the “Hello World” numbers must take into account the use of Gunicorn, with three work-ers, as detailed in Appendix A. The “Shortest Path” numbers, however, are gained withthe CMB test-server process running with a single thread, single process, making itsbehavior completely serial when multiple backends communicate with it.

66

8.3 Discussion

We speculate that this fact may be why there’s such a big difference in the parallelizationspeedup in Subtables 8.6a and 8.6b. Attempts to raise the GUNICORN_WORKER_TIMEOUTvalue in cmb-flask/source/scripts/gunicorn_start.sh to 150 seconds, but that didnot have any effect on the problems mentioned in Subsection 7.5.1, as this was our bestguess as to what could cause these errors from occurring.

Table 8.6: The speedup of the average runtimes per set in tests 4 and 5, divided by dev3 ’sBenchmark tests’ average timings per set.

Nth set Avgdev12(Nx)/Avgdev3(Nx) Avgdev123(Nx)/Avgdev3(Nx)1 1.520 1.0072 2.038 1.6073 2.422 2.3384 2.634 2.6525 3.137 3.3126 3.281 3.1287 3.350 3.5498 3.260 3.7039 3.825 3.86410 3.035 4.36611 3.708 4.41412 4.087 4.303

(a) “Hello World”

Nth set Avgdev12(Nx)/Avgdev3(Nx) Avgdev123(Nx)/Avgdev3(Nx)1 1.202 1.2092 1.541 1.4613 1.527 1.7314 1.589 1.7155 1.600 1.6746 1.743 1.7817 1.701 1.8798 1.714 1.8609 1.746 1.96910 1.749 2.02011 1.781 2.00312 1.806 1.970

(b) “Shortest Path”

Figure 8.5 show the trends of the numbers listed in Subtables 8.6a and 8.6b. The trianglesdenote values of the rightmost column of these two subtables, and the filled circles denotethe values from the middle column.

67

Chapter 8. Results

Albeit the fact that the variance in the numbers for “Hello World”, the trend in Figure 8.5is quite apparent; The “Hello World” tests, utilizing Gunicorn on the test-server, havenot stabilized with the 12 submissions of set N = 12 enough to give an idea as to thegraphs asymptote.

The “Shortest Path” lines in Figure 8.5 do not depict trends equally positive. However,we consider the omission of running tests 4’s and 5’s “Shortest Path” submissions withoutGunicorn to be an important point to keep in mind.

In the attempts to perform them with Gunicorn, results like Avgdev12(N = 12) = 280.70seconds were achieved, with 1 (one) submission reported as “failed” by the CMB test-server’s big-tests-results check.

Re-calculating the bottom middle number of Subtable 8.6b with this result, gives aspeedup of 2.228, which is demonstrably more positive when it comes to the capabili-ties of the proposed system implementation’s dispatcher than 1.806.

1 2 3 4 5 6 7 8 9 10 11 12N number of submissions in set

0.81.01.21.41.61.82.02.22.42.62.83.03.23.43.63.84.04.24.4 Hello World tests

Shortest Path tests

Figure 8.5: Parallelization speedup trends from tests’ averages.

With the results reported herein, we conclude that the proposed system implementationcan achieve a lower bound of 1.5x increased speedup when using two Odroid-XU3 backendsinstead of one, and can potentially approach an upper bound of > 4x better speedup underideal circumstances.

Keep in mind that the numbers used to get the numbers contained in Table 8.6 and

68

8.3 Discussion

depicted in Figure 8.5 come from the slowest backend of the test-system, dev3 .

Any system with faster backends will potentially be able to gain even better speedup withthis fixed workload.

8.3.4 Parallel Hypothesis

Hypoth. III The Dispatcher will have the effect that for N simultaneous, identical submissions(of the same problem) submitted to the CMB test-system, with M backends pollingthe server for submissions to execute, and with;a) 1 ≤ N ≤M , the total time the CMB system requires from the first upload until

last submission’s profiling is reported as completed, will at the most approachthe upper bound duration of 1 (one) analogous submission of said problem.

b) 1 ≤ M ≤ N and N mod M = 0, the total time the CMB system requiresfrom the first upload until last submission’s profiling is reported as completed,will at the most approach the upper bound duration of N

M analogous seriallyexecuted submissions of said problem.

c) 1 ≤M < N , the total time the CMB system requires from the first upload untillast submission’s profiling is reported as completed, will at the most approachthe upper bound duration of dNM e analogous serially executed submissions ofsaid problem.

What Hypothesis III proposes is that with the proposed system implementation, we shouldsee a decrease in the time necessitated by the system per uploaded submission, for eachbackend, added, and that the decrease should approximate the expressions in a), b), andc).

The first subpart of Hypothesis II; a), only says that for a single submission, and with mul-tiple backends polling the CMB system, the system should never take longer to completethe submission, than it would with just one backend polling it.

dev1 has the overall fastest execution of the serial tests, and it completed “Shortest Path”N = 1 in 95.18 seconds, while both of tests 4 and 5 cleared it faster than that.

However, the timing results for “Hello World” do not support Hypothesis IIIa), but insteadshow us, with the rest of the measurements depicted in Figure 8.3a, that there’s a lot ofvariance in the required execution time for problem submissions which execute fast, forthe proposed system implementation.

The case that Hypothesis IIIa) still holds true, but cannot be confirmed due to thevariances in timing with the proposed system implementation can be made. This isfurther discussed in Section 9.4.

Nevertheless, Hypothesis IIIb) is supported by the values depicted in the figures, and listedin the tables of this chapter. At no point, does a set with N (Nmod 3 = 0→ [3, 6, 9, 12])

69

Chapter 8. Results

submissions take longer in tests 4 or 5, than they do in any of the serial Benchmark tests.We thus consider the evidence presented to support Hypothesis IIIb).

Finally, what Hypothesis IIIc) attempts to cover, is that which a) and b) have not, thatthe time required for the system in these configurations to complete N submissions (whenNmod 3 6= 0), does not need more time than it would by adding 1 or 2 more submissions,making Nmod 3 = 0. In other words, that the system does not spike multitudes of thetime increment an additional submission would incur, when Nmod 3 6= 0.

With the exception of the results from first set N = 1 of tests 4 and 5 “Hello World” tests,which we’ve already stated we suspect vary so much due to the variances in the timingrequired by the CMB system (aside from the time required by the actual submission), weconsider the evidence given in this chapter to support Hypothesis IIIc).

We feel safe making that claim, due to none of the remaining sets N ∈ [2, 12] everapproaching the time it took the fastest backend dev1 to execute “Hello World” withN = 2.

70

Chapter 9Future Work

In this chapter, we discuss all the things there was insufficient time or capacity to ac-complish, in addition to any suggestions we feel worthwhile to consider for the future ofCMB.2

First we discuss the completion of the Automatic Monitoring and Recovery system im-plementation, the first contribution introduced in Section 1.2, before continuing withdiscussing potential improvements for the second contribution introduced in Section 1.2,the Dispatcher.

After that, we discuss expanding the functionality of the proposed system implementationso as to support multiple language problems/submissions. Finally, we discuss the rest ofthe non-Problem Statement introduced improvements we would like to propose that futuredevelopers of CMB keep in mind when continuing the development of this system.

9.1 Completing the Automatic SystemMonitoring andRecovery implementation

As already briefly touched upon in Section 4.2, the work with the automatic monitor-ing and recovery system was not completed due to the time constraints incurred by theevents described in the Preface. However, it was always our intent to implement a ro-bust system for this project goal, using the likes of Systemd (Poettering et al., 2010) orUpstart (Upstart, 2006).

Listing 9.1 shows the crontab script running every 15 minutes, which is how the CMBsystem currently (and at the outset of this project) performs its “automatic system re-

71

Chapter 9. Future Work

covery”.

Listing 9.1: The CMB crontab script used for monitoring the server processes of CMB, at theoutset of this project.

1 #!/bin/bash2 . $APPLICATION_SETTINGS3 lines=‘ps aux | grep "push.py" | wc −l‘4 sendMail=false5 msg=""6 pushDown=07 if [[ $lines == 1 ]]; then8 sendMail=true9 msg="PUSH␣DOWN!"10 pushDown=111 fi1213 guniDown=014 lines=‘ps aux | grep "gunicorn" | wc −l‘15 if [[ $lines == 1 ]]; then16 sendMail=true17 msg="$msg␣GUNICORN␣DOWN!"18 guniDown=119 fi2021 #check board22 ping −q −w 2 $BOARD_IP > /dev/null23 test=$?24 boardDown=025 if [[ $test != 0 ]]; then26 sendMail=true27 msg="$msg␣BOARD␣DOWN"28 boardDown=129 fi3031 obj="{\"pushDown\":$pushDown,␣\"guniDown\":$guniDown,␣\"boardDown\":

$boardDown,␣\"date\":\"‘date‘\"}"32 echo $obj >> /srv/climber/cmb/server/cmb−flask/logs/systemStability.txt3334 #send mail is done by crontab35 if [[ $sendMail == true ]]; then36 cd /srv/climber/cmb/frontend37 /usr/local/bin/gulp maintenance > /dev/null38 echo $msg39 fi

72

9.1 Completing the Automatic System Monitoring and Recovery implementation

There are a few critiques worth mentioning as pitfalls to avoid in a future implementationof this script for the CMB system:

1. Their use of the Linux terminal command grep to regex after the process namesgiven by the ps aux command, runs the risk of giving false positives.• This is detailed in a post on stackoverflow.com, which has an analogous useof grep : http://stackoverflow.com/a/3510850/1503549

2. The hard-coded location and file-name of where to save the JSON object createdfor logging purposes at the end of each execution of the script.• It is our opinion that these things belong in a Machine-specific environmentvariable, as discussed in Subsection 4.1.1.

3. The omnipresent (and mandatory for the successful execution of this script) use ofthe APPLICATION_SETTINGS environment variable.

Listing 9.2 shows how far the efforts regarding automatic monitoring in the proposedsystem implementation came, and the concept/premise we intend for its future use.

Listing 9.2: The proposed system implementation of the automatic monitoring of backends.290 @backend_routes.route(’/api/backends/status/’, methods=[’GET’])291 @backend_routes.route(’/api/backends/status/<int:optional_time_limit>/’, methods=[’GET’

])292 @myWrappers.sameIP_required293 def check_last_query_of_backends(optional_time_limit=20):294 backends = Backend.query.all()295296 late_backends = [("backend.id", "backend.last_query")]297 should_have_polled_since = datetime.now()298 should_have_polled_since −= timedelta(seconds=optional_time_limit)299 for backend in backends:300 # Late_backend = namedtuple(’backend_id’, ’last_query’)301 if (backend.last_query is None or302 backend.last_query < should_have_polled_since):303 # late_backends += Late_backend(backend.id, backend.last_query)304 late_backends += [(backend.id, backend.last_query)]305306 # If one or more backends are late:307 if len(late_backends) > 1:308 busy_backends = []309 for backend in late_backends:310 runs_assigned_to_backend = Run.query.filter(311 Run.backend_assigned == backend,312 Run.dequeued.isnot(None),313 Run.finished.is_(None)).first()314 if runs_assigned_to_backend:315 busy_backends += backend.id316317 return jsonify(318 returncode=1, late_backends=late_backends,

73

Chapter 9. Future Work

319 time_limit="{} second(s).".format(optional_time_limit),320 current_time_and_date="Current time and date on server: "321 "{}".format(getCurrentTimeString()),322 backends_currently_busy_with_a_code_profiling=busy_backends,323 message="Some backends have not polled within the given time limit.")324325 # If no backends were late:326 return jsonify(327 returncode=0,328 time_limit="{} second(s).".format(optional_time_limit),329 current_time_and_date="Current time and date on server: "330 "{}".format(getCurrentTimeString()),331 message="All backends have polled within the given time limit.")

As can be seen in Listing 9.2, it is incomplete in its functionality, but the tools for creatinga more thorough check of which backends are unresponsive are in place. In addition, it isvery easy, as exemplified in almost all of the Python files in cmb-flask/source/routesand cmb-board/service.py, to implement an independent Python script running every15 minutes (like the crontab script of Listing 9.1), and report by mail or otherwise to theadministrators if something is amiss.

This, combined with an implementation like what Systemd (Poettering et al., 2010) offersto ensure that the Flask web server’s process(es) are always running, would constituteboth an automatic monitoring and recovery implementation.

9.2 Future improvements to the Dispatcher

The Dispatcher, as implemented in the proposed system implementation, simply assignsthe next submission to be profiled to the first backend polling which happens to be eligible.

Thus, it’s merely acting as a First-In, First-Out (FIFO) queue handler, with no regardsto load-balancing. As such, there’s no code ensuring erroneous behavior, like the samebackend repeatedly polling while the others are silent, is handled correctly (besides what’smentioned in this chapter).

Hence, as more backends are added, especially backends with differing performance, codewhich helps load-balancing the system better than FIFO should be simple enough for anydeveloper familiar with the code base to implement, given the proposed system imple-mentation changes to both how the backends poll the Flask web server REST API fornew submissions to profile, and the database changes suggested in Section 4.4.

74

9.2 Future improvements to the Dispatcher

9.2.1 Expanding the dispatcher into a broker

One desire stated in the Problem Statement is for suggestions on how the Dispatchercould be expanded into a broker.

As previously stated, with the proposed system implementation, it is our opinion thatthere’s little need for a broker. Not only is every backend (with the exception of what’sdescribed in Sections 4.6 and 7) identical, but with the current “Master Makefile” (detailedin Subsection 9.3.1) being used in every compilation of every uploaded submission, it’sensured that (insofar as the server and backends remain constant/identical) nothing isdifferent between any two identical profiling executions.

However, if some backends at some point in the future are “put aside” for special purposes,or a change is made so that a problem may be run on different architectures as long asthe software required is present, then a broker may be of use.

The changes to the database schema, and the code deciding the behavior the CMBsystem with regards to backends should be simple enough to expand upon for anydeveloper familiar with the code base. Especially considering how consolidated thedatabase schema and backend behavior is in cmb-flask/source/database/models.pyand cmb-flask/source/routes/backends.py, respectively.

9.2.2 Discovering the upper limit of backends a server can handle

Parallel to expanding the Dispatcher into a broker, an additional point of future interestfor the Dispatcher is finding out the limit after which adding a new backend does nothelp improve the concurrency capabilities of the CMB system.

This, however, ties closely in with the implementation of the server, which with thediffering threads of Gunicorn (given that there are sufficient processing cores availablefor the Gunicorn “worker” threads), can more easily be parallelized, as long as importantsections of code is written so as to hinder race-conditions, such as shown in Listing 9.3.

It is from line 45 and until line 67 the atomic lock in Listing 9.3 ensures that no Gunicorn“worker” thread/process executes the code within simultaneously with another. The race-condition avoided by this implementation is to avoid that a scheduled run, not yet assignedto a backend, gets assigned to two backends so that later the Flask web servers RESTAPI will receive POST requests from both backends having run the same submission.

Listing 9.3: How the proposed system implementation handles the potential race-condition ofmultiple backends polling for submissions from the web server running with several Gunicorn“worker” threads.

29 atomic_db_access_lock = Lock() # Used to ensure atomic retrieval of potential runs30 backend_routes = Blueprint(’backend_routes’, __name__, None)

75

Chapter 9. Future Work

313233 def check_for_available_runs_by_backend(backend, LIFO_order=False):34 # Check first if a run is already assigned to backend35 already_assigned_run = Run.query.filter(36 Run.backend_assigned == backend,37 Run.dequeued.isnot(None),38 Run.finished.is_(None)).first()3940 if already_assigned_run is not None:41 return already_assigned_run4243 # If not, see if there’s a new one to assign44 untaken_runs = []45 atomic_db_access_lock.acquire() # Acquire mutex lock46 if LIFO_order:47 untaken_runs = Run.query.\48 filter_by(backend_assigned=None).\49 order_by(desc(Run.enqueued)).all()50 else:51 untaken_runs = Run.query.\52 filter_by(backend_assigned=None).\53 order_by(Run.enqueued).all()5455 for run in untaken_runs:56 if (run.submission.problem.architecture == backend.architecture and57 run.submission.problem.software_required <= set(backend.softwares)):58 # Update run that it’s going to get processed59 run.backend_assigned = backend60 run.dequeued = datetime.now()61 db.session.add(run)62 db.session.commit()63 atomic_db_access_lock.release() # Release mutex lock64 return run6566 # If no run was found in for−loop:67 atomic_db_access_lock.release() # Release mutex lock68 return None

As parting thoughts on the discussion of this subsection, it is our opinion that the inher-ently serial check of whether or not an uploaded submission successfully compiles shouldbe delegated to the backends, instead of the server.

This notion is spawned due to the premise to moving any inherently serial work the“single-point-of-contact” (the Flak web server) between all the elements of the CMBsystem performs, to the backends, which with proposed system implementation of thisreport there may be many attempting concurrent communication with the Flask webserver.

76

9.2 Future improvements to the Dispatcher

Amdahl’s Law of parallelization (Mark D. Hill and Michael R. Marty, 2008) foretells thisto be a potential choke point for future scalability of the CMB system with the currentbehavior.

9.2.3 Fixing the undiscovered Gunicorn bug

Listing 9.4 shows lines 147−191 and 216−252 of cmb-flask/source/routes/backends.py,which handle the case when the submission profiled on a backend is returned, and eitherfailed at some point on the backend or fails the final big correctness test on the server.

Listing 9.4: The lines of Python code where we believe the Gunicorn bug can occur.147 # Deal with the case that it didn’t finish successfully on backend148 if not json_data[’was_profiling_successfully_completed?’]:149 submission_of_run.state = "failed"150 current_run.msg = json_data[’msg’]151152 check_step = ProfilingStepReady(0)153 # While−loop for future readability, even though154 # it would’ve been "faster" to just create the155 # ProfilingStepReady corresponding to the value of156 # json_data[’last_successful_step’] instead...157 while check_step < json_data[’last_successful_step’]:158 check_step += 1159160 failed_step = check_step + 1161 print(getCurrentTimeString() + "Profiling failed during step {} on backend.".format(

failed_step))162 # If the failed step is during checking correctness of small tests,163 if failed_step > 4:164 submission_of_run.msg = (165 "Profiling of uploaded code failed on "166 "backend between ’{}’ and ’{}’".format(167 check_step.status, failed_step.status))168 else:169 submission_of_run.msg = current_run.msg170171 # Save and report172 db.session.add(backend)173 db.session.add(current_run)174 db.session.add(submission_of_run)175 db.session.commit()176 response = jsonify(message="Ack")177 response.status_code = 200178 return response179180 # Copy over tmpfile with big test(s)’s output from backend,181 # and save file in local folder cmb−flask/problems/<problem_name>/problemIO/

77

Chapter 9. Future Work

182 remote_path = json_data[’output_file_path’]183 local_path = secure_filename(current_run.submission.problem.name).lower()184 local_path += "/problemIO/{}_solution.txt".format(run_id)185 local_path = op.join(op.expandvars(os.environ[’UPLOAD_FOLDER’]), local_path)186 try:187 cmb_ssh_client = connectSSHClient(188 hostname=backend.ip_address, port=22,189 username=os.environ[’CMB_USER’], private_key_pw=os.environ[’

CMB_SERVER_SSH_PW’])190 print(getCurrentTimeString() + "\tMade SSH connection client.")216 except Exception as e:217 error_message = "\tFollowing Exception occured during SSH/SCP Operations:\n"218 error_message += "\t{}".format(e)219 print(getCurrentTimeString() + error_message)220 finally:221 print(getCurrentTimeString() + "\tClosing SSH connection client.")222 cmb_ssh_client.close()223224 if not op.exists(local_path):225 # Should never happen...226 print("\n\tFILE ’{}’ WAS NOT SUCCESSFULLY SCP’ED FROM BACKEND:\

n\t\t{}\n".format(remote_path, backend))227228 # Run checker executable (which should be located in directory to which the229 # tempfile was copied).230 os.chdir(op.dirname(local_path))231 execute_chckr_cmd = ["./checker", "input.txt", local_path, "answer.txt"]232 execute_chckr_dict = execute_os_command(cmd=execute_chckr_cmd, timeout=None)233234 # Parse checker output, see if output passed it or not235 submission_of_run.state = "finished"236 chckr_output = execute_chckr_dict[’stdout’].split("\n")237 if (execute_chckr_dict[’returncode’] != 0 and str(chckr_output[0]).lower() != "ok"):238 print(getCurrentTimeString() + "Profiling big test(s) output failed correctness test.")239 submission_of_run.msg = "Failed big correctness test"240 submission_of_run.state = "failed"241242 # Save and report243 backend.last_query = datetime.now()244 db.session.add(backend)245 db.session.add(current_run)246 db.session.add(submission_of_run)247 db.session.commit()248 response = jsonify(message="Ack")249 response.status_code = 200250 return response251252 os.remove(local_path)

78

9.3 Expanding CMB to support language-specific problems/submissions

We show this code here, to show the reader the only place (and what happens just before)in the proposed system implementation, the feedback message “Failed big correctness test”is set. This is the error which is shown when logging into the web interface of the test-system, after running the Flask web server with Gunicorn and more than one backendpolling the server of the test-system.

For any reader with experience reading Python code, it should be evident that there isnothing inherently serial with the code on lines 230− 250 of Listing 9.2.3. As such, theselines of code should offer no challenge for Gunicorn workers to execute, no matter whatamount of sibling-workers are active, as long as each worker handles the received HTTPrequest alone.

As Chapter 7 specifies, the tests are run with only “Hello World” or “Shortest Path”problem submissions between each database/system re-set, and each N ∈ [1, 12] set sub-mits 78 identical uploads of said problem submission. Yet, 1 − 3 of every 12 sets of 78“Shortest Path” test submissions fail with the “Failed big correctness test” error.

Thus, since this bug only occurs with “Shortest Path” submissions when the Flask webserver is run with Gunicorn, we do not consider it to be likely that the root cause of thisbug occurs on the backend.

However, we consider this a rather critical issue to consider for future work on the proposedCMB system implementation of this report.

9.3 Expanding CMB to support language-specific prob-lems/submissions

Section 4.4 proposes changes to the database, to easily facilitate the future support forsubmissions written in other programming languages than C/C++ to be uploaded. Thepremise is based on the fact that the administrators of the CMB system must updatethe database, letting it know what software is present on what backend, and throughthe software entered into the database, tell the system which backends supports whichprogramming languages.

Additonally, the proposed scheme detailed in Figure 4.1 also facilitates the relationshipbetween a Problem and the permitted languages with which to upload submissions forsaid Problem, through the SoftwareSet table/relationship with the Problem table. Onetip for future developers of the CMB system is to look into how the “abc” Python modulecan be of help: https://docs.python.org/3/library/abc.html

79

Chapter 9. Future Work

9.3.1 Permitting problem creators to edit C/C++ Makefile

One challenge not detailed in this report, which should also involve any developers workingon the front-end user-interface of CMB, is the adding a “Makefile template” for C/C++Problems in the CMB system. In the past year, it has often been the wish and desire ofmultiple parties interested in the CMB system to modify what is currently a “Master”Makefile, located in cmb-flask/Makefile. This Makefile is used both to test whether ornot a submission compiles on the server, in addition to compile and execute the uploadedsubmission code on the backends.

As such, it’d be more ideal if some effort could be spent on discovering what compila-tion parameters are mandatory for a successful profiling of an uploaded submission, andthrough the admin’s web-interface permit CMB administrators to modify the remainingparameters of the Makefile to their desire.

Examples of such differences of desire could be the use of OpenCL vs OpenMP. Or evenin a future implementation of CMB, MPI. However, this would require the storing of thestrings of which the different Makefiles consist in a central location, such as the databasecurrently in use by the CMB system, on a per-problem basis.

9.4 Remaining future potential improvements

This section details our remaining thoughts on what aspects of the CMB system futuredevelopers of the CMB system ought to keep in mind. Both for the sake of makingtheir efforts in developing the system simpler and easier, but also for the robustness andpotential future-proofing of the CMB system.

9.4.1 Folder re-structuring

The ulterior motive behind the changes of the proposed system implementation describedin Subsection 4.1.1, is to not only consolidate the configuration files/variables of the CMBsystem into a simple-to-find location but also to provide example as to how the rest ofthe system might be structured.

For instance, that the configs folder should be located in the top-level root-folder ofboth the cmb-board and cmb-flask Git repositories. (Likewise with the scripts folder).

As such, it’s our recommendation that new folders, and files, are not added in “happen-stance” locations within the Git repositories mentioned in this report, but that instead asemblance of common structure is strived for.

80

9.4 Remaining future potential improvements

9.4.2 Adding new architectures/backends to the proposed systemimplementation

With the detailing of the proposed changes to the backends from Section 4.6, we want tostress that if it ever becomes relevant to add software support for executing profiling ofuploaded submissions on a different type of backend/architecture, the cmb-board folderstructures is designed with that intent in mind.

The import statement at the top of cmb-board/service.py can be switched out withdifferent import statements, as long as the Python code containing the new code followthe below two guidelines:

1. The Python code supporting the profiling on a different architecture/backend islocated in a folder parallel to the cmb-flask/odroid_xu3 folder.

2. The Python code supporting the profiling on a different architecture/backend, con-tains the same function names/analogous steps, as the ones located in the cmb-flask/odroid_xu3, so that the only change required is the switching of the import state-ments at the top of the cmb-board/service.py file.

9.4.3 Improving and completing the DB schema in a future-proofingmanner

As stated in Section 4.4, we were unable to complete the intended changes for the databaseschema, especially regarding the relationship between the Software and Backend table,which we intended to be replaced by a relationship between the SoftwareSet and Backendtable, as illustrated in Figure 4.1.

Our main struggle with implementing the necessitated changes to the DB schema is thatthere seems to be a lot of misleading, if not erroneous documentation online, as to howPython code utilizing SQLAlchemy should be written/implemented.

Thus, it is our recommendation for any future developers of the CMB system, to spendthe time and effort to learning the difference between implementing Python SQL codestatements the Flask way, vs. the SQLAlchemy way. In our efforts, we experienced timeand time again, that the numerous, easy-to-find, examples and tutorials showing howto write SQLAlchemy code in a Flask Python project, often made things more difficult,instead of facilitating the changes desired.

This is also something we have noticed being a recurring issue, in much of the cmb-flask/source/*.py code base, and is something we highly recommend future developers of theCMB system to keep in mind when developing CMB. Python is a language in which “theintention is that there’s preferably only one way to write code correctly” (Peters, 2016),as opposed to Perl, where the notion is that “there are many different ways to correctlysolve the problem” (Wall et al., 2000).

81

Chapter 9. Future Work

9.4.4 Combining the efforts of Sindre Magnussen and this project

As previously mentioned in this report, Sindre Magnussen, the author of (Magnussen,2015), also worked on his master in parallel with this project, on the CMB system.

While his master focused on the user-interface and user-experience of CMB, the focusof the master project was to implement changes permitting the CMB system to profilemultiple uploaded submissions concurrently. As it stands now, each master resides inseparate, divergent Git branches in the same Git repositories.

The Git repositories being cmb-flask, and cmb-board, links can be found in Chapters 5and 6. The proposed system implementation of this report, also includes the Git submod-ule cmb_utils1, introduced in Subection 4.3.1, while Sindre Magnussen’s master projectimplements the use of SocketIO into the communication between the Flask web serverand the frontend shown in Figure 4.2.

Ideally, at some point in the future, both our master projects’ efforts can be combined,and thus strengthening the CMB system as a whole.

Additionally, as final word on this subject, the Git commit history of this master project(in Git branch “test-chrischa-branch”), will show common history with the current “mas-ter” branch in the cmb-flask Git repository. This is a consequence of our early effortsto git pull changes made by Sindre Magnussen in the CMB project, for his master, intothe code base of this master project, so that a future merging of the two will be moreeasily facilitated.

The majority of the merge efforts were mainly done with regards to the database in-teractions in the Python code, in addition to the cmb-flask/source/server.py andcmb-flask/source/manager.py files.

9.4.5 Stabilizing time requirements of the CMB software

Another potential for future work we would like to mention is to minimize the variancein timing in the different steps of the Flask web server.

Figure 8.3a illustrates the problem very well. While the “Hello World” test submissiondoes nothing more than print “Hello World!”, all the submissions are identical, so a lineartrend like what’s depicted in Figure 8.3b is what we expected.

Potential areas of interest to research for these variances can be any of the followingnon-exhaustive list:

1. Database calls to the MySQL database, located on another machine and communi-1 Whose URL endpoint is https://bitbucket.org/climbingmontblanc/cmb_utils/.

82

9.4 Remaining future potential improvements

cated with via the LAN network at IDI, NTNU.

2. Idiosyncrasies of Gunicorn with the proposed system implementation.

3. Uncontrollable delays in the live and otherwise-in-use LAN networks at IDI, NTNU.

4. Weird OS behavior on CMB server or backends, enforcing the use slower memoryother than the RAM memory, which in our tests have been shown to have sufficientcapacity.

Of the above four potential areas, we consider the first two to be the most likely oneswhere improvements can be found. The above subsection on improving the DB schemacan be kept in mind when considering improving (and making more effective use of) callsto the DB for data in the Python code on the server (cmb-flask).

9.4.6 Improving server storage efficiency

Multiple times during the execution of tests for Chapter 8, the VM offered as CMB test-server by IDI’s Technical Group got filled up when it approached the 12 to 14 hundredsof uploaded submissions.

It was this that first prompted us to write scripts to easily facilitate the reset of theDB between tests, and deleting all previously uploaded submission files on the CMBtest-server.

However, considering that this is a platform that has been used for mandatory assign-ments counting towards the final grade of University-level courses, and is currently beingevaluated as a platform used for C/C++ programming exams for students, we want tosuggest a hybrid system.

The hybrid system we envision would unpack and unzip all uploads, automatically removeOS X file system files and other similarly irrelevant files (this could be a hardcoded orperiodically/annually-/biannually-updated list), and generate a checksum for all the filesin the uploaded submission, much like Git.

Then, if a user submits the same files over and over, e.g. to get averaged energy mea-surements from the system, the CMB system could just re-utilize the same zip, ignoringthe ones the user upload after the first one, due to storing and repeatedly referring to thechecksum and the upload that originally created it.

Likewise, this checksum could then also be used to check whether any users (students)upload the same zip, from different accounts.

Additionally, if done right, it could perhaps be possible for the checksum to alert the CMBsystem how similar two uploads are. If this could be achieved, it could help warning the

83

Chapter 9. Future Work

creators of the problem in the CMB system of the submissions whose code is very similar,so that they perhaps warrant an extra close, manual look.

9.4.7 Coverage testing

Finally, again due to the time constraints made by the events described in the Preface,we were unable to write sufficient tests to achieve a test coverage of at minimum 90% ofthe added Python code.

During the efforts made in this project of upgrading the Python code base from Python 2to Python 3, the existing unit/integration tests were (all but one - the one integration testwritten by (Magnussen, 2015) -), working too, and this was after the making (splitting) ofthe cmb_utils folder into a Git submodule. Thus, most of the code now in the submodule,with the exception of some of the recursive code, was also covered by the tests (with acoverage higher than 90%).

As a concluding remark, it is our belief that there is a good chance (though not 100%certain), that the test’s code coverage could have increased to at least 75% of the Pythoncode in cmb-flask, if this project could have continued for another 4 to 6 weeks as statedin the Preface.

Writing unit tests for the Python code in the cmb-board code, may, however, be verydifficult, if the goal is to achieve > 90% code coverage with these. It’s our suspicion thatit may very well be more trouble than it’s worth, to attempt this goal without relying onimplementation tests, as opposed to unit tests.

84

Chapter 10Conclusion and contribution

This report has documented, described, and proposed a new system implementation forthe CMB system, so as to minimize the user-experienced wait-time when there are mul-tiple submissions submitted to the CMB Flask web server, in a short period of time.

The proposed system implementation has been tested with three backends, tests, andbackends both described in Chapter 7. Chapter 8 lists and discusses the results of thetests, and hypotheses detailed in Chapter 7.

Our test results from Chapter 8 show that our proposed system implementation doesindeed improve the throughput of the amount of submissions the CMB system is ableto handle concurrently, with parallelization speedup of two “Shortest Path” submissionsgiving ' 1.461 speedup, and ten submissions giving 2.020 speedup, when having threebackends in use by the test-system. Likewise, measured parallelization speedups rangedfrom 1.541 speedup with two “Shortest Path” submissions using the two fastest backends,up to 1.806 speedup achieved with twelve “Shortest Path” submissions.

These results, though hindered by the troubles of having submissions that should runthrough the system with the system reporting them as correct instead of erroneous whentest-server is executed with Gunicorn, has left us with the belief that a future effort toimplement the CMB system which can scale almost linearly with the available resources,can be achieved without gargantuan efforts.

Additionally, this effort could be of great value to the CMB system, especially if becomesmore heavily used in the future. It is our opinion that the CMB system could be veryuseful in the quest for facilitating the search for more energy efficient software and algo-rithms, but the CMB system does have its areas of potential improvement, as discussedin Chapter 9.

85

Chapter 10. Conclusion and contribution

10.1 Contribution

This report details several contributions to the CMB project under Lasse Natvig:

1. The implementation of a Dispatcher, permitting more than one backend to concur-rently profile an uploaded problem submission.(Described in Section 4.5).

• And thus re-writing the majority of the bash-scripts causing errors on theCMB system as it was at the outset of this project, into Python 3 code, so asto handle errors and exceptions more smoothly.

2. The upgrade of the code base to Python 3, from Python 2.(Described in Section 4.3).

3. The re-structuring of sensitive/secrets such as IP addresses and passwords, as wellas the “tidying up” of where these and any other environment variables are located.This to more readily permit current and future developers of the CMB system toquickly find the environment variables (and their data) that they need, and for themto know where new ones should be put.(Described in Subsection 4.1.1).

4. Simplified the start-up for cmb-flask, and created a similarly simple start-up forcmb-board. Also added a simplified start-up sequence for the frontend of the CMBsystem, for development mode.(Shown in Sections 5.3 and 6.3).

86

Bibliography

Barrett, D. J., Silverman, R. E., 2001. SSH, The Secure Shell: The Definitive Guide.O’Reilly & Associates, Inc., Sebastopol, CA, USA.

BSC, 2011. The Mont Blanc project. https://www.montblanc-project.eu/, accessedJan. 21st 2016.

Bull, 2016. An Atos brand for technology products and software. http://www.bull.com/,accessed Jan. 21st 2016.

Fu, N., Li, Y., Liu, B., Xu, H., Zhang, Y., Nov 2015. Realization of controlling emmc 5.0device based on fpga for automatic test system. In: IEEE AUTOTESTCON, 2015. pp.251–255.

Gajda, W., 2015. Pro Vagrant, 1st Edition. Apress, Berkely, CA, USA.

Green, B., Seshadri, S., 2013. AngularJS, 1st Edition. O’Reilly Media, Inc.

Gunicorn, 2010. Gunicorn, An Open-Source Project. http://gunicorn.org/, accessedMar. 22nd 2016.

help.ubuntu.com/community/, 2015. SSH/TransferFiles. Accessed Jul. 6th 2016.

help.ubuntu.com/community/, 2016. The Cron Daemon System. https://help.ubuntu.com/community/CronHowto, accessed Jul. 2nd 2016.

http://www.crowdsourcing.com/, 2016. http://www.crowdsourcing.com/. http://www.crowdsourcing.com/, accessed Jul. 7th 2016.

Kattis, 2016. Kattis. http://www.kattis.com/, accessed Jul. 6th 2016.

Magnussen, S., 2015. Improvements, Stability and Handover of Climbing Mont Blanc.

Mark D. Hill and Michael R. Marty, 2008. Amdahl’s Law in the Multicore Era. IEEEComputer 41, 33–38.

87

BIBLIOGRAPHY

Merkel, D., Mar. 2014. Docker: Lightweight Linux Containers for Consistent Developmentand Deployment.URL http://dl.acm.org/citation.cfm?id=2600239.2600241

Natvig, L., Follan, T., Støa, S., Magnussen, S., García-Guirado, A., 2015. Climbing MontBlanc - A Training Site for Energy Efficient Programming on Heterogeneous MulticoreProcessors. CoRR abs/1511.02240.URL http://arxiv.org/abs/1511.02240

Peters, T., 2016. The Zen of PythonAccessed Jul. 7th 2016.

Poettering, L., Sievers, K., Hoyer, H., Gundersen, D. M. T., Herrmann, D., 2010. Upstart,An Open-Source Project. https://freedesktop.org/wiki/Software/systemd/, ac-cessed Mar. 22nd 2016.

Qian, Y., 2012. Automatic parallelization tools. In: Proceedings of the World Congresson Engineering and Computer Science. Vol. 1.

Rai, R., 2013. Socket. IO Real-time Web Application Development. Packt Publishing Ltd.

Rajovic, N., Carpenter, P. M., Gelado, I., Puzovic, N., Ramirez, A., Valero, M., 2013.Supercomputing with commodity cpus: Are mobile socs ready for hpc? In: Proceedingsof the International Conference on High Performance Computing, Networking, Storageand Analysis. SC ’13. ACM, New York, NY, USA, pp. 40:1–40:12.URL http://doi.acm.org/10.1145/2503210.2503281

Ronacher, A., 2010. Flask, a Python micro web-framework. (Open-Source Project). http://flask.pocoo.org/, accessed Mar. 22nd 2016.

Solem, A., 2009. Celery, a distributed Task based event queue. (Open-Source Project).http://celery.readthedocs.org/en/, accessed Mar. 22nd 2016.

Støa, S., Follan, T., June 2015. Climbing Mont Blanc. Master’s thesis, Department ofComputer and Information Science (IDI), Norwegian University of Science and Tech-nology (NTNU), Norway.

Subramaniam, B., Feng, W.-c., 2010. Understanding Power Measurement Implications inthe Green500 List. In: Proceedings of the 2010 IEEE/ACM Int’L Conference on GreenComputing and Communications & Int’L Conference on Cyber, Physical and SocialComputing. GREENCOM-CPSCOM ’10. IEEE Computer Society, Washington, DC,USA, pp. 245–251.URL http://dx.doi.org/10.1109/GreenCom-CPSCom.2010.140

Upstart, 2006. systemdd, An Open-Source Project. http://upstart.ubuntu.com/, ac-cessed Mar. 22nd 2016.

Wall, L., Christiansen, T., Orwant, J., 2000. Programming Perl - there’s more than oneway to do it (3. ed.). O’Reilly.

www.hardkernel.com, 2016. ODROID-XU3. http://www.hardkernel.com/main/products/prdt_info.php?g_code=g140448267127, accessed Jun. 30th 2016.

88

BIBLIOGRAPHY

www.sdcard.org, 2016. Bus Speed (Default Speed/ High Speed/ UHS). https://www.sdcard.org/developers/overview/bus_speed/index.html, accessed Jun. 30th2016.

YouTube, 2016. YouTube. https://www.youtube.com/results?search_query=Celery+Framework, accessed Jul. 7th 2016.

ZeroMQ, 2011. ZeroMQ. http://zeromq.org/bindings:python, accessed Mar. 21st2016.

89

BIBLIOGRAPHY

90

Appendices

91

Appendix ATest Set-Up Configs

This appendix contains commented copies of the configurations files used for the testsreported in this thesis, and the “source scripts” used to source the variables in the con-figuration files before starting the CMB processes.

First, we give the “source scripts” of the server and the backends, to give the user aperspective of how the configurations are all connected. Thereafter, we list and commenton the different configuration files, starting with the files containing the “secret” environ-ment variables for the server and backends, before finishing with the “machine specific”environment variables for the server and backends.

92

A.1 Source Scripts Files

A.1 Source Scripts Files

A.1.1 Server Source Script File

Listing A.1: Environment variables source-script of test-server.

1 #! /bin/bash23 set −a45 # Find out where this file is located:6 THIS_FILE="${BASH_SOURCE[0]}"7 while [ −h "$THIS_FILE" ]; do # resolve $THIS_FILE until the file is no longer a

symlink8 DIR="$(␣cd␣−P␣"$( dirname "$THIS_FILE" )"␣&&␣pwd␣)"9 THIS_FILE="$(readlink␣"$THIS_FILE")"10 [[ $THIS_FILE != /∗ ]] && THIS_FILE="$DIR/$THIS_FILE" # if

$THIS_FILE was a relative symlink, we need to resolve it relative to thepath where the symlink file was located

11 done1213 # Always assume that structure is cmb/cmb−flask/scripts, cmb/cmb−flask/configs14 cur_dir=$(pwd)15 CMB_DIR=$(realpath "$cur_dir/$(dirname␣$THIS_FILE)/../..")16 CMB_FLASK_CONFIGS_DIR="$CMB_DIR/cmb−flask/configs"1718 # Source passwords, usernames, secrets...19 SECRETS="$CMB_FLASK_CONFIGS_DIR/test−secrets.cfg"20 source $SECRETS2122 # Needs CMB_DIR variable set(!), and secrets sourced!!!23 MACHINE_SETTINGS="$CMB_FLASK_CONFIGS_DIR/test−server.cfg"24 source "$MACHINE_SETTINGS"

93

Chapter A. Test Set-Up Configs

A.1.2 Backends Source Script File

Listing A.2: Environment variables source-script of backends.

1 #!/bin/bash23 set −a45 # Find out where this file is located:6 THIS_FILE="${BASH_SOURCE[0]}"7 while [ −h "$THIS_FILE" ]; do # resolve $THIS_FILE until the file is no longer a

symlink8 DIR="$(␣cd␣−P␣"$( dirname "$THIS_FILE" )"␣&&␣pwd␣)"9 THIS_FILE="$(readlink␣"$THIS_FILE")"10 [[ $THIS_FILE != /∗ ]] && THIS_FILE="$DIR/$THIS_FILE" # if

$THIS_FILE was a relative symlink, we need to resolve it relative to thepath where the symlink file was located

11 done1213 # Always assume that structure is cmb−board/scripts14 cur_dir=$(pwd)15 CMB_DIR=$(realpath "$cur_dir/$(dirname␣$THIS_FILE)/..")1617 # Source passwords, usernames, secrets...18 source "$CMB_DIR/configs/test−secrets.cfg"1920 # Needs CMB_DIR variable set(!)21 source "$CMB_DIR/configs/odroid−xu3.cfg"

94

A.2 Secret Environment Variable(s) Config Files

A.2 Secret Environment Variable(s) Config Files

A.2.1 Server Secrets Config File

Listing A.3: Secret/sensitive environment variables of test-server.

1 # /bin/bash23 CMB_USER="<redacted␣Ubuntu␣OS␣user␣name>"4 CMB_USER_GROUP="<redacted␣Ubuntu␣OS␣user␣group>"56 CMB_SERVER_SSH_PW="<redacted␣pw>"7 BACKEND_SHARED_PW="<redacted␣pw>"89 CMB_MAIL_PASSWORD="<redacted␣pw>"10 CMB_MAIL_USERNAME="<redacted␣email>"1112 CMB_SECRET_KEY="<redacted␣pw>"13 CMB_TOKEN_SECRET="<redacted␣pw>"1415 CMB_MYSQL_USER="<redacted␣user␣name>"16 CMB_MYSQL_PASSWORD="<redacted␣pw>"17 CMB_MYSQL_DATABASE="cmb_dev−tests"

A.2.2 Backends Secrets Config File

Listing A.4: Secret/sensitive environment variables of backends.

1 BACKEND_ID=<redacted id #>2 BACKEND_SSH_PW="<redacted␣pw>"3 BACKEND_SHARED_PW="<redacted␣pw>"45 CMB_USER="<redacted␣Ubuntu␣OS␣user␣name>"6 CMB_SERVER_PORT=<redacted port #>7 CMB_SERVER_SSH_PORT=<redacted port #>8 CMB_SERVER="<redacted␣url>"

95

Chapter A. Test Set-Up Configs

A.3 Machine-Specific Environment Variable(s) Config Files

A.3.1 Server Specific Config File

Listing A.5: Machine-specific environment variables of test-server.

1 VERSION="dev"2 SERVER_PORT=<redacted port #>3 GUNICORN_LOG_LEVEL="debug"45 CMB_SERVER="0.0.0.0"6 BOARD_IPs="\7 <redacted␣ip>,\8 <redacted␣ip>,\9 <redacted␣ip>"10 CMB_MYSQL_SERVER="mysql.idi.ntnu.no"1112 MAIL_PORT=<redacted port #>13 MAIL_USE_SSL=True14 MAIL_USE_TLS=False15 MAIL_SERVER="smtp.gmail.com"1617 FLASK_DIR="$CMB_DIR/cmb−flask"18 FLASK_VENV_DIR="$FLASK_DIR/venv"19 FRONTEND_DIR="$CMB_DIR/cmb−frontend"20 MALI_DIR="$CMB_DIR/Mali_OpenCL_SDK_v1.1.0"2122 CMB_LOG_FOLDER="$FLASK_DIR/logs"23 UPLOAD_FOLDER="$FLASK_DIR/problems"2425 SQLALCHEMY_DATABASE_URI="mysql://$CMB_MYSQL_USER:

$CMB_MYSQL_PASSWORD@$CMB_MYSQL_SERVER/$CMB_MYSQL_DATABASE"

96

A.4 Test-Server Gunicorn start-script/config

A.3.2 Backends Specific Config File

Listing A.6: Machine-specific environment variables of backends.

1 VERSION="dev"23 BACKEND_DIR="/home/climber/cmb−board/"45 timeoutLength=906 targetTemperature=607 samplingInterval=100008 temperatureFile="/sys/devices/10060000.tmu/temp"

A.4 Test-Server Gunicorn start-script/config

Listing A.7: Test-server’s Gunicorn start-script/config.

1 #!/bin/bash23 # Find out where this file is located:4 THIS_FILE="${BASH_SOURCE[0]}"5 while [ −h "$THIS_FILE" ]; do # resolve $THIS_FILE until the file is no longer a

symlink6 DIR="$(␣cd␣−P␣"$( dirname "$THIS_FILE" )"␣&&␣pwd␣)"7 THIS_FILE="$(readlink␣"$THIS_FILE")"8 [[ $THIS_FILE != /∗ ]] && THIS_FILE="$DIR/$THIS_FILE" # if $THIS_FILE

was a relative symlink, we need to resolve it relative to the path where thesymlink file was located

9 done1011 set −e12 source $(dirname $THIS_FILE)/test−source_cmb_envvars.sh1314 NUM_WORKERS=315 NAME="CMB−Flask"16 mkdir −p $CMB_LOG_FOLDER17 GUNICORN_LOG="$CMB_LOG_FOLDER/gunicorn.log"1819 # Necessary due to the time it takes for checker.cpp to compile and check big test20 # results on shortest path timing out the default timeout of gunicorn workers21 GUNICORN_WORKER_TIMEOUT=1202223 echo "Starting␣$NAME"24

97

Chapter A. Test Set-Up Configs

25 # Start unicorn26 GUNICORN_CONFIG=$FLASK_DIR/configs/test−gunicorn.cfg27 if [[ ! −f $GUNICORN_CONFIG ]]; then28 $FLASK_VENV_DIR/bin/gunicorn manager:app −b $CMB_SERVER:

$SERVER_PORT \29 −−preload \30 −−name $NAME \31 −−workers $NUM_WORKERS \32 −−chdir=$FLASK_DIR/source \33 −−timeout $GUNICORN_WORKER_TIMEOUT \34 −−user=$CMB_USER −−group=$CMB_USER_GROUP \35 −−log−syslog−prefix="$NAME␣Gunicorn" \36 −−log−level=$GUNICORN_LOG_LEVEL \37 −−access−logfile $GUNICORN_LOG \38 −−log−file=−39 else40 # Not implemented due to file not existing41 exit 142 fi

98

Appendix BSSH Install and Set-Up Note

The way the proposed system implementation of this thesis’s project (Chapter 4) hasbeen implemented, has been with care towards security, stability, and feasibility for futureenhancements.

Thus, when having to decide upon how to transfer files between the backends and serverof the CMB system, the author of this thesis chose to utilize SSH (Barrett and Silverman,2001). However, to uphold the care mentioned in the previous paragraph, the author ofthis project decided to have each backend make its own private/public RSA key pair,and have said key pair require a password to unlock/use them, with the password beingunique to each key/pair.

With that solution in mind, each of the backends had their own unique password storedin cmb-board/configs/*secrets.cfg, and did not need to share any common passwordfor allowing transfer of files between the backends and server (given that ssh−copy−idhas been used as instructed in Chapters 5 and 6). Hence, each backend could make use ofSSH to connect to the server without having access to anything but their own passwordunlocking its key pair, and likewise the server could connect to each of the backends, forboth the purpose of remote command execution, as well as the purpose of file transferbetween machines.

The code installed through the instructions given in Chapter 6, specifies how to generatethese private/public RSA key pairs, and the dialog that shows up prompts the user to adda password for the utilization of the new key-par. This password is contained within theBACKEND_SSH_PW environment variable, given in line 3 of Listing A.2.2, and is requiredfor the proposed system implementation of Chapter 4 to work.

Hence, after the creation of the key pair has been completed at the server and each

99

Chapter B. SSH Install and Set-Up Note

backend, it is recommended to follow the ssh−copy−id steps in Chapters 5 and 6.If the backends are unable to retrieve the files for a submission they have been givento profile from the server after this, it is recommended to first attempt the equivalentSCP (help.ubuntu.com/community/, 2015) commands, and attempt to the decipher theroot of the problem through this venue.

If the SCP does not luck out (in other words; it works fine with these commands), whatcan penultimately be attempted, is to execute the Python code manually, line by line,through the command venv/bin/ipython , which should be installed if the instructionsof Chapter 6 have been followed.

The final (though some argue this perhaps should be first, not final) venue to attempt isto perform the SCP from a terminal on the backend, adding −v flag to the command,for more debugging output. The −v flag can be appended up to −vvv , for maximumdebug output. Conversely, there should be a log on the server specifying what wentwrong in the authorization/connection process, if it happened server-side (or the receiverif the problem is from the server to the backend). The typical location of this log on therecipient machine is /var/log/auth.log.

100

Appendix CTest-VM Specifications

In this appendix, we list the full commands (and outputs), from which the key stats listedin Chapter 7 to describe the CMB test-server VM, was extracted.

First we list the commands and outputs for the OS and Kernel before we finish with theCPU and Memory.

C.1 OS and Kernel Information

Listing C.1: OS and Kernel information of test-server.

1 test−user@test−vm:~$ lsb_release −a2 No LSB modules are available.3 Distributor ID: Ubuntu4 Description: Ubuntu 16.04 LTS5 Release: 16.046 Codename: xenial7 test−user@test−vm:~$ uname −a8 Linux test−vm 4.4.0−22−generic #40−Ubuntu SMP Thu May 12 22:03:46 UTC 2016

x86_64 x86_64 x86_64 GNU/Linux9 test−user@test−vm:~$

101

Chapter C. Test-VM Specifications

C.2 CPU Information

Listing C.2: CPU information of test-server.

1 test−user@test−vm:~$ lscpu2 Architecture: x86_643 CPU op−mode(s): 32−bit, 64−bit4 Byte Order: Little Endian5 CPU(s): 36 On−line CPU(s) list: 0−27 Thread(s) per core: 18 Core(s) per socket: 39 Socket(s): 110 NUMA node(s): 111 Vendor ID: GenuineIntel12 CPU family: 613 Model: 4514 Model name: Intel(R) Xeon(R) CPU E5−2650 0 @ 2.00GHz15 Stepping: 716 CPU MHz: 1999.99817 BogoMIPS: 3999.9918 Hypervisor vendor: Microsoft19 Virtualization type: full20 L1d cache: 32K21 L1i cache: 32K22 L2 cache: 256K23 L3 cache: 20480K24 NUMA node0 CPU(s): 0−225 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36

clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl eagerfpupni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lmxsaveopt

26 test−user@test−vm:~$ cat /proc/cpuinfo27 processor : 028 vendor_id : GenuineIntel29 cpu family : 630 model : 4531 model name : Intel(R) Xeon(R) CPU E5−2650 0 @ 2.00GHz32 stepping : 733 microcode : 0xffffffff34 cpu MHz : 1999.99835 cache size : 20480 KB36 physical id : 037 siblings : 338 core id : 039 cpu cores : 3

102

C.3 Memory Information

40 apicid : 041 initial apicid : 042 fpu : yes43 fpu_exception : yes44 cpuid level : 1345 wp : yes46 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36

clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl eagerfpupni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lmxsaveopt

47 bugs :48 bogomips : 3999.9949 clflush size : 6450 cache_alignment : 6451 address sizes : 42 bits physical, 48 bits virtual52 power management:5354 <!!! Two identical duplicate processors removed for brevity !!>5556 test−user@test−vm:~$

C.3 Memory Information

Listing C.3: Memory information of test-server.

1 test−user@test−vm:~$ sudo blockdev −−getsize64 /dev/sda2 161061273603 test−user@test−vm:~$ cat /proc/meminfo4 MemTotal: 2338692 kB5 MemFree: 544796 kB6 MemAvailable: 1325384 kB7 Buffers: 302720 kB8 Cached: 393988 kB9 SwapCached: 31080 kB10 Active: 652472 kB11 Inactive: 250932 kB12 Active(anon): 74500 kB13 Inactive(anon): 152740 kB14 Active(file): 577972 kB15 Inactive(file): 98192 kB16 Unevictable: 0 kB17 Mlocked: 0 kB18 SwapTotal: 2095100 kB19 SwapFree: 1950868 kB

103

Chapter C. Test-VM Specifications

20 Dirty: 80 kB21 Writeback: 0 kB22 AnonPages: 204388 kB23 Mapped: 35804 kB24 Shmem: 20544 kB25 Slab: 149600 kB26 SReclaimable: 127284 kB27 SUnreclaim: 22316 kB28 KernelStack: 4208 kB29 PageTables: 11236 kB30 NFS_Unstable: 0 kB31 Bounce: 0 kB32 WritebackTmp: 0 kB33 CommitLimit: 3264444 kB34 Committed_AS: 784248 kB35 VmallocTotal: 34359738367 kB36 VmallocUsed: 0 kB37 VmallocChunk: 0 kB38 HardwareCorrupted: 0 kB39 AnonHugePages: 24576 kB40 CmaTotal: 0 kB41 CmaFree: 0 kB42 HugePages_Total: 043 HugePages_Free: 044 HugePages_Rsvd: 045 HugePages_Surp: 046 Hugepagesize: 2048 kB47 DirectMap4k: 114624 kB48 DirectMap2M: 2375680 kB49 test−user@test−vm:~$

104

Appendix DBackends Specifications

In this appendix, we first list all the commands and the full output of the key stats usedto describe OS, Kernel, CPU and Memory specifications of the three backends used inthe tests of this project.

Thereafter, we list all installed OS packages on the three backends, dev1, dev2, and dev3.First, we list the ones installed on all three, before we, in turn, list the packages installedon both of a pair, in the order as specified by the headings that follow.

Thus, the results from Chapter 8 can be reproduced, since with the packages installedon all three backends, and the packages installed on each pair-wise combination of thebackends, it should be straightforward to find which backend had which packages installed.

D.1 OS and Kernel Specifications of Backends

D.1.1 Backend 1

Listing D.1: OS and Kernel information of backend dev1.

1 test−user@<redacted>−odroid−xu3−dev1:~:$ lsb_release −a2 No LSB modules are available.3 Distributor ID: Ubuntu4 Description: Ubuntu 14.04.4 LTS5 Release: 14.046 Codename: trusty7 test−user@<redacted>−odroid−xu3−dev1:~:$ uname −a

105

Chapter D. Backends Specifications

8 Linux <redacted>−odroid−xu3−dev1 3.10.54+ #1 SMP PREEMPT Wed Sep 1014:01:26 UTC 2014 armv7l armv7l armv7l GNU/Linux

9 test−user@<redacted>−odroid−xu3−dev1:~:$

D.1.2 Backend 2

Listing D.2: OS and Kernel information of backend dev2.

1 test−user@<redacted>−odroid−xu3−dev2:~$ lsb_release −a2 No LSB modules are available.3 Distributor ID: Ubuntu4 Description: Ubuntu 14.04.4 LTS5 Release: 14.046 Codename: trusty7 test−user@<redacted>−odroid−xu3−dev2:~$ uname −a8 Linux <redacted>−odroid−xu3−dev2 3.10.69 #1 SMP PREEMPT Thu Feb 12

15:22:14 BRST 2015 armv7l armv7l armv7l GNU/Linux9 test−user@<redacted>−odroid−xu3−dev2:~$

D.1.3 Backend 3

Listing D.3: OS and Kernel information of backend dev3.

1 test−user@<redacted>−odroid−xu3−dev3:~$ lsb_release −a2 No LSB modules are available.3 Distributor ID: Ubuntu4 Description: Ubuntu 15.105 Release: 15.106 Codename: wily7 test−user@<redacted>−odroid−xu3−dev3:~$ uname −a8 Linux <redacted>−odroid−xu3−dev3 3.10.96−78 #1 SMP PREEMPT Fri Feb 12

05:59:25 BRST 2016 armv7l armv7l armv7l GNU/Linux9 test−user@<redacted>−odroid−xu3−dev3:~$

D.2 CPU Specifications of Backends

D.2.1 Backend 1

Listing D.4: CPU information of backend dev1.

106

D.2 CPU Specifications of Backends

1 test−user@<redacted>−odroid−xu3−dev1:~:$ lscpu2 Architecture: armv7l3 Byte Order: Little Endian4 CPU(s): 85 On−line CPU(s) list: 0−76 Thread(s) per core: 17 Core(s) per socket: 48 Socket(s): 29 test−user@<redacted>−odroid−xu3−dev1:~:$ cat /proc/cpuinfo10 processor : 011 model name : ARMv7 Processor rev 3 (v7l)12 BogoMIPS : 84.0013 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt14 CPU implementer : 0x4115 CPU architecture: 716 CPU variant : 0x017 CPU part : 0xc0718 CPU revision : 31920 <!!! Three identical duplicate processors removed for brevity !!>2122 processor : 423 model name : ARMv7 Processor rev 3 (v7l)24 BogoMIPS : 120.0025 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt26 CPU implementer : 0x4127 CPU architecture: 728 CPU variant : 0x229 CPU part : 0xc0f30 CPU revision : 33132 <!!! Three identical duplicate processors removed for brevity !!>333435 Hardware : ODROID−XU336 Revision : 000037 Serial : 000000000000000038 test−user@<redacted>−odroid−xu3−dev1:~:$

D.2.2 Backend 2

Listing D.5: CPU information of backend dev2.

107

Chapter D. Backends Specifications

1 test−user@<redacted>−odroid−xu3−dev2:~$ lscpu2 Architecture: armv7l3 Byte Order: Little Endian4 CPU(s): 85 On−line CPU(s) list: 0−76 Thread(s) per core: 17 Core(s) per socket: 48 Socket(s): 29 test−user@<redacted>−odroid−xu3−dev2:~$ cat /proc/cpuinfo10 processor : 011 model name : ARMv7 Processor rev 3 (v7l)12 BogoMIPS : 84.0013 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt14 CPU implementer : 0x4115 CPU architecture: 716 CPU variant : 0x017 CPU part : 0xc0718 CPU revision : 31920 <!!! Three identical duplicate processors removed for brevity !!>2122 processor : 423 model name : ARMv7 Processor rev 3 (v7l)24 BogoMIPS : 120.0025 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt26 CPU implementer : 0x4127 CPU architecture: 728 CPU variant : 0x229 CPU part : 0xc0f30 CPU revision : 33132 <!!! Three identical duplicate processors removed for brevity !!>3334 Hardware : ODROID−XU335 Revision : 000036 Serial : 000000000000000037 test−user@<redacted>−odroid−xu3−dev2:~$

D.2.3 Backend 3

Listing D.6: CPU information of backend dev3.

1 test−user@<redacted>−odroid−xu3−dev3:~$ lscpu2 Architecture: armv7l

108

D.3 Memory Specifications of Backends

3 Byte Order: Little Endian4 CPU(s): 85 On−line CPU(s) list: 0−76 Thread(s) per core: 17 Core(s) per socket: 48 Socket(s): 29 Model name: ARMv7 Processor rev 3 (v7l)10 CPU max MHz: 1400.000011 CPU min MHz: 200.000012 test−user@<redacted>−odroid−xu3−dev3:~$ cat /proc/cpuinfo13 processor : 014 model name : ARMv7 Processor rev 3 (v7l)15 BogoMIPS : 84.0016 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt17 CPU implementer : 0x4118 CPU architecture: 719 CPU variant : 0x020 CPU part : 0xc0721 CPU revision : 32223 <!!! Three identical duplicate processors removed for brevity !!>2425 processor : 426 model name : ARMv7 Processor rev 3 (v7l)27 BogoMIPS : 120.0028 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt29 CPU implementer : 0x4130 CPU architecture: 731 CPU variant : 0x232 CPU part : 0xc0f33 CPU revision : 33435 <!!! Three identical duplicate processors removed for brevity !!>3637 Hardware : ODROID−XU338 Revision : 010039 Serial : 000000000000000040 test−user@<redacted>−odroid−xu3−dev3:~$

D.3 Memory Specifications of Backends

D.3.1 Backend 1

109

Chapter D. Backends Specifications

Listing D.7: Memory information of backend dev1.

1 test−user@<redacted>−odroid−xu3−dev1:~:$ sudo blockdev −−getsize64 /dev/mmcblk0

2 312685363203 test−user@<redacted>−odroid−xu3−dev1:~:$ cat /proc/meminfo4 MemTotal: 2043084 kB5 MemFree: 1288048 kB6 Buffers: 156436 kB7 Cached: 85240 kB8 SwapCached: 0 kB9 Active: 156880 kB10 Inactive: 120244 kB11 Active(anon): 35480 kB12 Inactive(anon): 3200 kB13 Active(file): 121400 kB14 Inactive(file): 117044 kB15 Unevictable: 0 kB16 Mlocked: 0 kB17 HighTotal: 1296384 kB18 HighFree: 1032004 kB19 LowTotal: 746700 kB20 LowFree: 256044 kB21 SwapTotal: 0 kB22 SwapFree: 0 kB23 Dirty: 28 kB24 Writeback: 0 kB25 AnonPages: 35528 kB26 Mapped: 14804 kB27 Shmem: 3220 kB28 Slab: 312656 kB29 SReclaimable: 289120 kB30 SUnreclaim: 23536 kB31 KernelStack: 2128 kB32 PageTables: 1116 kB33 NFS_Unstable: 0 kB34 Bounce: 0 kB35 WritebackTmp: 0 kB36 CommitLimit: 1021540 kB37 Committed_AS: 497808 kB38 VmallocTotal: 245760 kB39 VmallocUsed: 18152 kB40 VmallocChunk: 103580 kB41 test−user@<redacted>−odroid−xu3−dev1:~:$

110

D.3 Memory Specifications of Backends

D.3.2 Backend 2

Listing D.8: Memory information of backend dev2.

1 test−user@<redacted>−odroid−xu3−dev2:~$ sudo blockdev −−getsize64 /dev/mmcblk0

2 312685363203 test−user@<redacted>−odroid−xu3−dev2:~:$ cat /proc/meminfo4 MemTotal: 2043108 kB5 MemFree: 1331516 kB6 Buffers: 128052 kB7 Cached: 298596 kB8 SwapCached: 0 kB9 Active: 360736 kB10 Inactive: 128956 kB11 Active(anon): 65572 kB12 Inactive(anon): 740 kB13 Active(file): 295164 kB14 Inactive(file): 128216 kB15 Unevictable: 0 kB16 Mlocked: 0 kB17 HighTotal: 1296384 kB18 HighFree: 795480 kB19 LowTotal: 746724 kB20 LowFree: 536036 kB21 SwapTotal: 0 kB22 SwapFree: 0 kB23 Dirty: 40 kB24 Writeback: 0 kB25 AnonPages: 63072 kB26 Mapped: 15088 kB27 Shmem: 3264 kB28 Slab: 55112 kB29 SReclaimable: 33924 kB30 SUnreclaim: 21188 kB31 KernelStack: 1960 kB32 PageTables: 1292 kB33 NFS_Unstable: 0 kB34 Bounce: 0 kB35 WritebackTmp: 0 kB36 CommitLimit: 1021552 kB37 Committed_AS: 323728 kB38 VmallocTotal: 245760 kB39 VmallocUsed: 18168 kB40 VmallocChunk: 103580 kB41 test−user@<redacted>−odroid−xu3−dev2:~:$

111

Chapter D. Backends Specifications

D.3.3 Backend 3

Listing D.9: Memory information of backend dev3.

1 test−user@<redacted>−odroid−xu3−dev3:~$ sudo blockdev −−getsize64 /dev/mmcblk0

2 313220136963 test−user@<redacted>−odroid−xu3−dev3:~$ cat /proc/meminfo4 MemTotal: 2038540 kB5 MemFree: 1438556 kB6 Buffers: 25408 kB7 Cached: 308636 kB8 SwapCached: 0 kB9 Active: 276032 kB10 Inactive: 105268 kB11 Active(anon): 49936 kB12 Inactive(anon): 20684 kB13 Active(file): 226096 kB14 Inactive(file): 84584 kB15 Unevictable: 0 kB16 Mlocked: 0 kB17 HighTotal: 1296384 kB18 HighFree: 797032 kB19 LowTotal: 742156 kB20 LowFree: 641524 kB21 SwapTotal: 0 kB22 SwapFree: 0 kB23 Dirty: 0 kB24 Writeback: 0 kB25 AnonPages: 47292 kB26 Mapped: 13952 kB27 Shmem: 23364 kB28 Slab: 52348 kB29 SReclaimable: 31820 kB30 SUnreclaim: 20528 kB31 KernelStack: 1472 kB32 PageTables: 1268 kB33 NFS_Unstable: 0 kB34 Bounce: 0 kB35 WritebackTmp: 0 kB36 CommitLimit: 1019268 kB37 Committed_AS: 342936 kB38 VmallocTotal: 245760 kB39 VmallocUsed: 18740 kB40 VmallocChunk: 103580 kB41 test−user@<redacted>−odroid−xu3−dev3:~$

112

D.4 Packages installed on all three backends

D.4 Packages installed on all three backends1. accountsservice2. acl3. adduser4. alsa-base5. alsa-utils6. anacron7. apport8. apport-gtk9. apport-symptoms

10. apt11. aptdaemon12. aptdaemon-data13. apt-utils14. aria215. at-spi2-core16. autoconf17. automake18. autopoint19. autotools-dev20. avahi-daemon21. avahi-utils22. axel23. base-files24. base-passwd25. bash26. bash-completion27. bc28. bind9-host29. binutils30. blueman31. bluez32. bluez-cups33. bsdmainutils34. bsdutils35. build-essential36. busybox-initramfs37. bzip238. ca-certificates39. colord40. console-setup41. coreutils42. cpio43. cpp

44. cpp-4.945. cpp-546. cracklib-runtime47. cron48. cups49. cups-browsed50. cups-bsd51. cups-client52. cups-common53. cups-core-drivers54. cups-daemon55. cups-filters56. cups-filters-core-drivers57. cups-ppdc58. cups-server-common59. curl60. dash61. dbus62. dbus-x1163. dc64. dconf-cli65. dconf-gsettings-backend:armhf66. dconf-service67. debconf68. debconf-i18n69. debhelper70. debianutils71. desktop-file-utils72. dh-autoreconf73. dh-python74. dialog75. dictionaries-common76. diffstat77. diffutils78. dmsetup79. dmz-cursor-theme80. dnsmasq-base81. dpkg82. dpkg-dev83. e2fslibs:armhf84. e2fsprogs85. eject86. ethtool

113

Chapter D. Backends Specifications

87. evolution-data-server-common88. fail2ban89. ffmpegthumbnailer90. file91. findutils92. fontconfig93. fontconfig-config94. fonts-dejavu-core95. fonts-freefont-ttf96. fonts-kacst97. fonts-kacst-one98. fonts-khmeros-core99. fonts-lao100. fonts-lklug-sinhala101. fonts-sil-abyssinica102. fonts-sil-padauk103. fonts-takao-pgothic104. fonts-thai-tlwg105. fonts-tibetan-machine106. fonts-tlwg-garuda107. fonts-tlwg-kinnari108. fonts-tlwg-loma109. fonts-tlwg-mono110. fonts-tlwg-norasi111. fonts-tlwg-purisa112. fonts-tlwg-sawasdee113. fonts-tlwg-typewriter114. fonts-tlwg-typist115. fonts-tlwg-typo116. fonts-tlwg-umpush117. fonts-tlwg-waree118. foomatic-db-compressed-ppds119. fuse120. g++121. g++-4.9122. g++-5123. galculator124. gawk125. gcc126. gcc-4.9127. gcc-4.9-base:armhf128. gcc-5129. gcc-5-base:armhf130. gconf2131. gconf2-common

132. gconf-service133. gconf-service-backend134. gcr135. gdb136. genisoimage137. gettext138. gettext-base139. gfortran140. ghostscript141. ghostscript-x142. gir1.2-atk-1.0143. gir1.2-gnomekeyring-1.0144. gir1.2-gst-plugins-base-1.0145. gir1.2-gstreamer-1.0146. gir1.2-gtk-2.0147. gir1.2-notify-0.7148. gir1.2-packagekitglib-1.0149. gir1.2-soup-2.4150. git151. git-man152. gksu153. glib-networking:armhf154. glib-networking-common155. glib-networking-services156. glmark2-data157. glmark2-es2158. gnome-accessibility-themes159. gnome-desktop3-data160. gnome-icon-theme161. gnome-icon-theme-symbolic162. gnome-keyring163. gnome-menus164. gnome-themes-standard:armhf165. gnome-themes-standard-data166. gnome-user-guide167. gnupg168. gpgv169. grep170. groff-base171. gsettings-desktop-schemas172. gsfonts173. gstreamer1.0-clutter174. gstreamer1.0-plugins-base:armhf175. gstreamer1.0-plugins-good:armhf176. gstreamer1.0-pulseaudio:armhf

114

D.4 Packages installed on all three backends

177. gstreamer1.0-x:armhf178. gtk2-engines:armhf179. gtk2-engines-murrine:armhf180. gtk2-engines-pixbuf:armhf181. guvcview182. gvfs:armhf183. gvfs-backends184. gvfs-common185. gvfs-daemons186. gvfs-libs:armhf187. gzip188. hardening-includes189. hdparm190. hicolor-icon-theme191. hostname192. hplip193. hplip-data194. htop195. hunspell-en-ca196. hunspell-en-us197. hyphen-en-us198. ibus199. ibus-gtk3:armhf200. ibus-gtk:armhf201. ifupdown202. imagemagick-common203. im-config204. indicator-application205. initramfs-tools206. initramfs-tools-bin207. initscripts208. init-system-helpers209. inputattach210. insserv211. intltool-debian212. iproute2213. iptables214. iputils-arping215. iputils-ping216. isc-dhcp-client217. isc-dhcp-common218. iso-codes219. joe220. kbd221. kerneloops-daemon

222. keyboard-configuration223. klibc-utils224. kmod225. language-pack-en226. language-pack-en-base227. language-pack-gnome-en228. language-pack-gnome-en-base229. language-selector-common230. language-selector-gnome231. laptop-detect232. less233. libaa1:armhf234. libaacs0:armhf235. libaccountsservice0:armhf236. libacl1:armhf237. libandroid-properties1238. libapparmor1:armhf239. libappindicator3-1240. libapt-pkg-perl241. libarchive13:armhf242. libarchive-extract-perl243. libarchive-zip-perl244. libart-2.0-2:armhf245. libasan1:armhf246. libasan2:armhf247. libasn1-8-heimdal:armhf248. libasound2:armhf249. libasound2-data250. libasound2-plugins:armhf251. libassuan0:armhf252. libasyncns0:armhf253. libatasmart4:armhf254. libatk1.0-0:armhf255. libatk1.0-data256. libatk-bridge2.0-0:armhf257. libatlas3-base258. libatlas-base-dev259. libatlas-dev260. libatomic1:armhf261. libatspi2.0-0:armhf262. libattr1:armhf263. libaudio2:armhf264. libaudit1:armhf265. libaudit-common266. libauthen-sasl-perl

115

Chapter D. Backends Specifications

267. libavahi-client3:armhf268. libavahi-common3:armhf269. libavahi-common-data:armhf270. libavahi-core7:armhf271. libavahi-glib1:armhf272. libavc1394-0:armhf273. libbind9-90274. libblas3275. libblas-dev276. libblkid1:armhf277. libbluetooth3:armhf278. libbluray1:armhf279. libbsd0:armhf280. libburn4281. libbz2-1.0:armhf282. libc6:armhf283. libc6-dbg:armhf284. libc6-dev:armhf285. libcaca0:armhf286. libcairo2:armhf287. libcairo-gobject2:armhf288. libcanberra0:armhf289. libcanberra-gtk0:armhf290. libcanberra-gtk3-0:armhf291. libcap2:armhf292. libcap2-bin293. libc-ares2:armhf294. libc-bin295. libcc1-0:armhf296. libcddb2297. libc-dev-bin298. libcdio13299. libcdio-cdda1300. libcdio-paranoia1301. libcdparanoia0:armhf302. libcec303. libcgmanager0:armhf304. libchromaprint0:armhf305. libck-connector0:armhf306. libclass-accessor-perl307. libclone-perl308. libcloog-isl4:armhf309. libclutter-1.0-0:armhf310. libclutter-gst-2.0-0:armhf311. libclutter-gtk-1.0-0:armhf

312. libcolamd2.8.0:armhf313. libcomerr2:armhf314. libcrack2:armhf315. libcroco3:armhf316. libcups2:armhf317. libcupscgi1:armhf318. libcupsfilters1:armhf319. libcupsimage2:armhf320. libcupsmime1:armhf321. libcupsppdc1:armhf322. libcurl3:armhf323. libcurl3-gnutls:armhf324. libdatrie1:armhf325. libdb5.3:armhf326. libdbus-1-3:armhf327. libdbus-glib-1-2:armhf328. libdbusmenu-glib4:armhf329. libdbusmenu-gtk3-4:armhf330. libdbusmenu-gtk4:armhf331. libdc1394-22:armhf332. libdca0:armhf333. libdconf1:armhf334. libdebconfclient0:armhf335. libdevmapper1.02.1:armhf336. libdigest-hmac-perl337. libdirectfb-1.2-9:armhf338. libdjvulibre21:armhf339. libdjvulibre-text340. libdns100341. libdpkg-perl342. libdrm2:armhf343. libdrm-dev:armhf344. libdrm-exynos1:armhf345. libdrm-freedreno1:armhf346. libdrm-nouveau2:armhf347. libdrm-omap1:armhf348. libdrm-radeon1:armhf349. libdv4:armhf350. libdvdnav4:armhf351. libdvdread4:armhf352. libedit2:armhf353. libegl1-mesa:armhf354. libelf1:armhf355. libemail-valid-perl356. libenca0:armhf

116

D.4 Packages installed on all three backends

357. libenchant1c2a:armhf358. libencode-locale-perl359. liberror-perl360. libestr0361. libevent-2.0-5:armhf362. libexempi3:armhf363. libexif12:armhf364. libexpat1:armhf365. libexpat1-dev:armhf366. libfaad2:armhf367. libffi6:armhf368. libfftw3-double3:armhf369. libfftw3-single3:armhf370. libfile-basedir-perl371. libfile-copy-recursive-perl372. libfile-listing-perl373. libflac8:armhf374. libfontconfig1:armhf375. libfontembed1:armhf376. libfontenc1:armhf377. libfreetype6:armhf378. libfribidi0:armhf379. libfuse2:armhf380. libgail18:armhf381. libgbm1:armhf382. libgcc1:armhf383. libgcc-4.9-dev:armhf384. libgcc-5-dev:armhf385. libgck-1-0:armhf386. libgconf-2-4:armhf387. libgcr-3-common388. libgcr-base-3-1:armhf389. libgcr-ui-3-1:armhf390. libgd3:armhf391. libgdbm3:armhf392. libgdk-pixbuf2.0-0:armhf393. libgdk-pixbuf2.0-common394. libgeoclue0:armhf395. libgeoip1:armhf396. libgfortran3:armhf397. libgif4:armhf398. libgksu2-0399. libgl1-mesa-dri:armhf400. libgl1-mesa-glx:armhf401. libglapi-mesa:armhf

402. libgles1-mesa:armhf403. libgles2-mesa:armhf404. libglib2.0-0:armhf405. libglib2.0-bin406. libglib2.0-data407. libglu1-mesa:armhf408. libgmp10:armhf409. libgnome-keyring0:armhf410. libgnome-keyring-common411. libgnome-menu-3-0412. libgnutls-openssl27:armhf413. libgomp1:armhf414. libgpg-error0:armhf415. libgpgme11:armhf416. libgphoto2-6:armhf417. libgpm2:armhf418. libgpod4:armhf419. libgpod-common420. libgraphite2-3:armhf421. libgs9422. libgs9-common423. libgsm1:armhf424. libgssapi3-heimdal:armhf425. libgssapi-krb5-2:armhf426. libgssdp-1.0-3427. libgstreamer1.0-0:armhf428. libgstreamer-plugins-base1.0-0:armhf429. libgstreamer-plugins-good1.0-0:armhf430. libgtk2.0-0:armhf431. libgtk2.0-common432. libgtk-3-0:armhf433. libgtk-3-bin434. libgtk-3-common435. libgtop2-common436. libgudev-1.0-0:armhf437. libgupnp-1.0-4438. libgusb2:armhf439. libgutenprint2440. libgweather-common441. libgxps2:armhf442. libhardware2443. libharfbuzz0b:armhf444. libharfbuzz-icu0:armhf445. libhcrypto4-heimdal:armhf446. libheimbase1-heimdal:armhf

117

Chapter D. Backends Specifications

447. libheimntlm0-heimdal:armhf448. libhpmud0449. libhtml-parser-perl450. libhtml-tagset-perl451. libhtml-tree-perl452. libhttp-cookies-perl453. libhttp-date-perl454. libhttp-message-perl455. libhttp-negotiate-perl456. libhunspell-1.3-0:armhf457. libhx509-5-heimdal:armhf458. libhybris-common1459. libibus-1.0-5:armhf460. libice6:armhf461. libidn11:armhf462. libido3-0.1-0:armhf463. libiec61883-0:armhf464. libieee1284-3:armhf465. libimobiledevice4:armhf466. libindicator3-7467. libindicator7468. libio-html-perl469. libio-pty-perl470. libio-socket-inet6-perl471. libio-socket-ssl-perl472. libio-string-perl473. libipc-run-perl474. libipc-system-simple-perl475. libisc95476. libisccc90477. libisccfg90478. libiso9660-8479. libisofs6480. libiw30:armhf481. libjack-jackd2-0:armhf482. libjasper1:armhf483. libjavascriptcoregtk-3.0-0:armhf484. libjbig0:armhf485. libjbig2dec0486. libjpeg8:armhf487. libjpeg-turbo8:armhf488. libjs-jquery489. libjson-c2:armhf490. libjson-glib-1.0-0:armhf491. libjson-glib-1.0-common

492. libjte1493. libk5crypto3:armhf494. libkate1495. libkeyutils1:armhf496. libklibc497. libkmod2:armhf498. libkpathsea6499. libkrb5-26-heimdal:armhf500. libkrb5-3:armhf501. libkrb5support0:armhf502. liblapack3503. liblapack-dev504. liblcms2-2:armhf505. libldap-2.4-2:armhf506. libldb1:armhf507. liblightdm-gobject-1-0508. liblircclient0509. liblist-moreutils-perl510. liblocale-gettext-perl511. liblog-message-simple-perl512. liblqr-1-0:armhf513. libltdl7:armhf514. liblua5.2-0:armhf515. liblwp-mediatypes-perl516. liblwp-protocol-https-perl517. liblwres90518. liblzma5:armhf519. liblzo2-2:armhf520. libmad0:armhf521. libmagic1:armhf522. libmailtools-perl523. libmhash2:armhf524. libmicrohttpd10525. libmirclient-dev:armhf526. libmm-glib0:armhf527. libmnl0:armhf528. libmodule-pluggable-perl529. libmount1:armhf530. libmp3lame0:armhf531. libmpc3:armhf532. libmpdec2:armhf533. libmpeg2-4:armhf534. libmpfr4:armhf535. libmtdev1:armhf536. libmtp9:armhf

118

D.4 Packages installed on all three backends

537. libmtp-common538. libmtp-runtime539. libmysqlclient18:armhf540. libnatpmp1541. libnautilus-extension1a542. libncurses5:armhf543. libncursesw5:armhf544. libneon27-gnutls545. libnet-dbus-perl546. libnet-dns-perl547. libnet-domain-tld-perl548. libnetfilter-conntrack3:armhf549. libnet-http-perl550. libnet-ip-perl551. libnetpbm10552. libnet-smtp-ssl-perl553. libnet-ssleay-perl554. libnewt0.52:armhf555. libnfnetlink0:armhf556. libnih1:armhf557. libnih-dbus1:armhf558. libnl-3-200:armhf559. libnl-genl-3-200:armhf560. libnl-route-3-200:armhf561. libnm-gtk-common562. libnotify4:armhf563. libnspr4:armhf564. libnss3:armhf565. libnss3-nssdb566. libnss-mdns:armhf567. libogg0:armhf568. libopenobex1569. liborc-0.4-0:armhf570. libp11-kit0:armhf571. libp11-kit-gnome-keyring:armhf572. libpackagekit-glib2-16:armhf573. libpam0g:armhf574. libpam-gnome-keyring:armhf575. libpam-modules:armhf576. libpam-modules-bin577. libpam-runtime578. libpam-systemd:armhf579. libpango-1.0-0:armhf580. libpango1.0-0:armhf581. libpangocairo-1.0-0:armhf

582. libpangoft2-1.0-0:armhf583. libpangox-1.0-0:armhf584. libpangoxft-1.0-0:armhf585. libpaper1:armhf586. libpaper-utils587. libparse-debianchangelog-perl588. libpcap0.8:armhf589. libpci3:armhf590. libpciaccess0:armhf591. libpciaccess-dev:armhf592. libpcre3:armhf593. libpcsclite1:armhf594. libperlio-gzip-perl595. libpipeline1:armhf596. libpixman-1-0:armhf597. libpixman-1-dev598. libpng12-0:armhf599. libpod-latex-perl600. libpolkit-agent-1-0:armhf601. libpolkit-backend-1-0:armhf602. libpolkit-gobject-1-0:armhf603. libpoppler-glib8:armhf604. libpopt0:armhf605. libportaudio2:armhf606. libprocps3:armhf607. libprotobuf-dev:armhf608. libptexenc1609. libpthread-stubs0-dev:armhf610. libpulse0:armhf611. libpulsedsp:armhf612. libpulse-mainloop-glib0:armhf613. libpurple-bin614. libpwquality1:armhf615. libpwquality-common616. libpython2.7:armhf617. libpython2.7-dev:armhf618. libpython2.7-minimal:armhf619. libpython2.7-stdlib:armhf620. libpython3.4:armhf621. libpython3.4-dev:armhf622. libpython3.4-minimal:armhf623. libpython3.4-stdlib:armhf624. libpython3-dev:armhf625. libpython3-stdlib:armhf626. libpython-dev:armhf

119

Chapter D. Backends Specifications

627. libpython-stdlib:armhf628. libqt4-dbus:armhf629. libqt4-declarative:armhf630. libqt4-designer:armhf631. libqt4-dev632. libqt4-dev-bin633. libqt4-help:armhf634. libqt4-network:armhf635. libqt4-opengl:armhf636. libqt4-opengl-dev637. libqt4-qt3support:armhf638. libqt4-script:armhf639. libqt4-scripttools:armhf640. libqt4-sql:armhf641. libqt4-sql-mysql:armhf642. libqt4-svg:armhf643. libqt4-test:armhf644. libqt4-xml:armhf645. libqt4-xmlpatterns:armhf646. libqtcore4:armhf647. libqtdbus4:armhf648. libqtgui4:armhf649. libqtwebkit4:armhf650. libqtwebkit-dev651. libquvi7:armhf652. libquvi-scripts653. libqwt6654. libqwt-dev655. libraptor2-0:armhf656. librasqal3:armhf657. libraw1394-11:armhf658. librdf0:armhf659. libreadline6:armhf660. libroken18-heimdal:armhf661. librsvg2-2:armhf662. librsvg2-common:armhf663. libsamplerate0:armhf664. libsane:armhf665. libsane-common666. libsane-hpaio667. libsasl2-2:armhf668. libsasl2-modules:armhf669. libsasl2-modules-db:armhf670. libschroedinger-1.0-0:armhf671. libsdl1.2debian:armhf

672. libsdl-image1.2:armhf673. libsecret-1-0:armhf674. libsecret-common675. libselinux1:armhf676. libsemanage1:armhf677. libsemanage-common678. libsensors4:armhf679. libsepol1:armhf680. libsgutils2-2681. libshout3:armhf682. libsigsegv2:armhf683. libslang2:armhf684. libsm6:armhf685. libsmbclient:armhf686. libsndfile1:armhf687. libsnmp30:armhf688. libsnmp-base689. libsocket6-perl690. libsoup2.4-1:armhf691. libsoup-gnome2.4-1:armhf692. libspectre1:armhf693. libspeex1:armhf694. libspeexdsp1:armhf695. libsqlite3-0:armhf696. libss2:armhf697. libssh2-1:armhf698. libssh-4:armhf699. libssl1.0.0:armhf700. libstartup-notification0:armhf701. libstdc++-4.9-dev:armhf702. libstdc++-5-dev:armhf703. libstdc++6:armhf704. libsub-identify-perl705. libsub-name-perl706. libtalloc2:armhf707. libtasn1-6:armhf708. libtcl8.6:armhf709. libtdb1:armhf710. libterm-ui-perl711. libtevent0:armhf712. libtext-charwidth-perl713. libtext-iconv-perl714. libtext-levenshtein-perl715. libtext-soundex-perl716. libtext-wrapi18n-perl

120

D.4 Packages installed on all three backends

717. libthai0:armhf718. libthai-data719. libtheora0:armhf720. libtiff5:armhf721. libtimedate-perl722. libtinfo5:armhf723. libtk8.6:armhf724. libtool725. libubsan0:armhf726. libudev1:armhf727. libudisks2-0:armhf728. libunistring0:armhf729. liburi-perl730. libusb-0.1-4:armhf731. libusb-1.0-0:armhf732. libustr-1.0-1:armhf733. libutempter0734. libuuid1:armhf735. libv4l-0:armhf736. libv4lconvert0:armhf737. libva1:armhf738. libvdpau1:armhf739. libvisual-0.4-0:armhf740. libvorbis0a:armhf741. libvorbisenc2:armhf742. libvorbisfile3:armhf743. libvte9744. libvte-common745. libwavpack1:armhf746. libwayland-client0:armhf747. libwayland-cursor0:armhf748. libwayland-dev749. libwayland-egl1-mesa:armhf750. libwayland-server0:armhf751. libwbclient0:armhf752. libwebcam0753. libwebkitgtk-3.0-0:armhf754. libwebkitgtk-3.0-common755. libwebp5:armhf756. libwebpmux1:armhf757. libwhoopsie0758. libwind0-heimdal:armhf759. libwmf0.2-7:armhf760. libwnck22761. libwnck-3-0:armhf

762. libwnck-3-common763. libwnck-common764. libwrap0:armhf765. libwww-perl766. libwww-robotrules-perl767. libx11-6:armhf768. libx11-data769. libx11-dev:armhf770. libx11-xcb1:armhf771. libx11-xcb-dev:armhf772. libxau6:armhf773. libxau-dev:armhf774. libxaw7:armhf775. libxcb1:armhf776. libxcb1-dev:armhf777. libxcb-dri2-0:armhf778. libxcb-dri2-0-dev:armhf779. libxcb-dri3-0:armhf780. libxcb-dri3-dev:armhf781. libxcb-glx0:armhf782. libxcb-glx0-dev:armhf783. libxcb-icccm4:armhf784. libxcb-image0:armhf785. libxcb-keysyms1:armhf786. libxcb-present0:armhf787. libxcb-present-dev:armhf788. libxcb-randr0:armhf789. libxcb-randr0-dev:armhf790. libxcb-render0:armhf791. libxcb-render0-dev:armhf792. libxcb-shape0:armhf793. libxcb-shape0-dev:armhf794. libxcb-shm0:armhf795. libxcb-sync1:armhf796. libxcb-sync-dev:armhf797. libxcb-xfixes0:armhf798. libxcb-xfixes0-dev:armhf799. libxcb-xv0:armhf800. libxcomposite1:armhf801. libxcursor1:armhf802. libxdamage1:armhf803. libxdamage-dev:armhf804. libxdmcp6:armhf805. libxdmcp-dev:armhf806. libxext6:armhf

121

Chapter D. Backends Specifications

807. libxext-dev:armhf808. libxfixes3:armhf809. libxfixes-dev:armhf810. libxfont1:armhf811. libxft2:armhf812. libxi6:armhf813. libxinerama1:armhf814. libxkbcommon0:armhf815. libxkbcommon-dev816. libxkbfile1:armhf817. libxkbfile-dev:armhf818. libxklavier16819. libxml2:armhf820. libxml-parser-perl821. libxml-twig-perl822. libxmu6:armhf823. libxmuu1:armhf824. libxpm4:armhf825. libxrandr2:armhf826. libxrender1:armhf827. libxres1:armhf828. libxshmfence1:armhf829. libxshmfence-dev:armhf830. libxslt1.1:armhf831. libxss1:armhf832. libxt6:armhf833. libxtables10834. libxtst6:armhf835. libxv1:armhf836. libxvidcore4:armhf837. libxxf86dga1:armhf838. libxxf86vm1:armhf839. libxxf86vm-dev:armhf840. libyajl2:armhf841. libyelp0842. libzvbi0:armhf843. libzvbi-common844. lightdm-gtk-greeter845. lintian846. linux-firmware847. linux-libc-dev:armhf848. linux-sound-base849. lm-sensors850. locales851. login

852. logrotate853. lp-solve854. lsb-base855. lsb-release856. lshw857. m4858. make859. makedev860. man-db861. mawk862. mesa-utils863. mesa-utils-extra864. mime-support865. mobile-broadband-provider-info866. modemmanager867. module-init-tools868. mount869. mscompress870. multiarch-support871. mysql-common872. nautilus-data873. ncurses-base874. ncurses-bin875. netbase876. netcat-openbsd877. netpbm878. net-tools879. network-manager880. network-manager-gnome881. notification-daemon882. obex-data-server883. openprinting-ppds884. openssh-client885. openssh-server886. openssh-sftp-server887. openssl888. p11-kit889. p11-kit-modules:armhf890. parted891. passwd892. patch893. patchutils894. pciutils895. pcmciautils896. perl

122

D.4 Packages installed on all three backends

897. perl-base898. perl-modules899. pkg-config900. po-debconf901. policykit-1902. policykit-desktop-privileges903. poppler-data904. poppler-utils905. powermgmt-base906. ppp907. printer-driver-c2esp908. printer-driver-foo2zjs909. printer-driver-foo2zjs-common910. printer-driver-gutenprint911. printer-driver-hpcups912. printer-driver-min12xxw913. printer-driver-pnm2ppa914. printer-driver-postscript-hp915. printer-driver-ptouch916. printer-driver-pxljr917. printer-driver-sag-gdi918. printer-driver-splix919. procps920. psmisc921. pulseaudio922. pulseaudio-module-x11923. pulseaudio-utils924. python925. python2.7926. python2.7-dev927. python2.7-minimal928. python3929. python3.4930. python3.4-dev931. python3.4-minimal932. python3-apport933. python3-apt934. python3-aptdaemon935. python3-aptdaemon.gtk3widgets936. python3-aptdaemon.pkcompat937. python3-chardet938. python3-dbus939. python3-debian940. python3-defer941. python3-dev

942. python3-distupgrade943. python3-gi944. python3-minimal945. python3-pkg-resources946. python3-problem-report947. python3-pycurl948. python3-six949. python3-software-properties950. python3-update-manager951. python3-xkit952. python-apt953. python-apt-common954. python-cairo955. python-chardet956. python-chardet-whl957. python-colorama-whl958. python-crypto959. python-dbus960. python-dbus-dev961. python-decorator962. python-dev963. python-distlib-whl964. python-gi965. python-gobject966. python-gobject-2967. python-gtk2968. python-html5lib-whl969. python-imaging970. python-ldb971. python-minimal972. python-numpy973. python-pip-whl974. python-pkg-resources975. python-requests976. python-requests-whl977. python-samba978. python-scipy979. python-setuptools-whl980. python-six981. python-six-whl982. python-talloc983. python-tdb984. python-urllib3985. python-urllib3-whl986. python-virtualenv

123

Chapter D. Backends Specifications

987. qdbus988. qpdf989. qt4-default990. qt4-linguist-tools991. qt4-qmake992. qtchooser993. qtcore4-l10n994. quilt995. readline-common996. resolvconf997. rfkill998. rsync999. rsyslog1000. samba-common1001. samba-common-bin1002. samba-libs:armhf1003. sane-utils1004. screen1005. sed1006. sensible-utils1007. sgml-base1008. shared-mime-info1009. software-properties-common1010. software-properties-gtk1011. sound-theme-freedesktop1012. ssl-cert1013. strace1014. stress1015. sudo1016. system-config-printer-common1017. system-config-printer-gnome1018. system-config-printer-udev1019. systemd-shim1020. sysvinit-utils1021. sysv-rc1022. t1utils1023. tar1024. tcl1025. tcl8.61026. tex-common1027. texlive-binaries1028. time1029. tk1030. tk8.61031. transmission-common

1032. transmission-gtk1033. ttf-indic-fonts-core1034. ttf-ubuntu-font-family1035. tzdata1036. u-boot-tools1037. ubuntu-drivers-common1038. ubuntu-keyring1039. ubuntu-release-upgrader-core1040. ubuntu-release-upgrader-gtk1041. ucf1042. udev1043. udisks21044. ufw1045. unattended-upgrades1046. unzip1047. update-inetd1048. update-manager1049. update-manager-core1050. update-notifier1051. update-notifier-common1052. upower1053. usb-modeswitch1054. usb-modeswitch-data1055. usbmuxd1056. usbutils1057. util-linux1058. uvcdynctrl1059. uvcdynctrl-data1060. vim1061. vim-common1062. vim-runtime1063. vim-tiny1064. wamerican1065. wbritish1066. wget1067. whiptail1068. whois1069. whoopsie1070. wireless-tools1071. wpasupplicant1072. x11-apps1073. x11-common1074. x11proto-core-dev1075. x11proto-damage-dev1076. x11proto-dri2-dev

124

D.5 Packages installed on backend 1 and not 2

1077. x11proto-dri3-dev1078. x11proto-fixes-dev1079. x11proto-fonts-dev1080. x11proto-gl-dev1081. x11proto-input-dev1082. x11proto-kb-dev1083. x11proto-present-dev1084. x11proto-randr-dev1085. x11proto-render-dev1086. x11proto-resource-dev1087. x11proto-scrnsaver-dev1088. x11proto-video-dev1089. x11proto-xext-dev1090. x11proto-xf86bigfont-dev1091. x11proto-xf86dri-dev1092. x11proto-xf86vidmode-dev1093. x11proto-xinerama-dev1094. x11-session-utils1095. x11-utils1096. x11-xkb-utils1097. x11-xserver-utils1098. xauth1099. xbitmaps1100. xdg-user-dirs1101. xdg-utils1102. xfonts-base1103. xfonts-encodings1104. xfonts-scalable

1105. xfonts-utils1106. xinit1107. xinput1108. xkb-data1109. xml-core1110. xorg1111. xorg-docs-core1112. xorg-sgml-doctools1113. xserver-common1114. xserver-xorg1115. xserver-xorg-dev1116. xserver-xorg-input-all1117. xserver-xorg-input-evdev1118. xserver-xorg-input-synaptics1119. xserver-xorg-video-all1120. xserver-xorg-video-fbdev1121. xterm1122. xtrans-dev1123. xutils-dev1124. xz-utils1125. yelp1126. yelp-xsl1127. zenity1128. zenity-common1129. zip1130. zlib1g:armhf1131. zlib1g-dev:armhf

D.5 Packages installed on backend 1 and not 21. aglfn2. aptitude3. aptitude-common4. apt-xapian-index5. chromium-browser6. chromium-browser-l10n7. clinfo8. docbook-utils9. firefox-locale-af

10. firefox-locale-ar11. firefox-locale-bg12. firefox-locale-ca13. firefox-locale-cy

14. firefox-locale-da15. firefox-locale-de16. firefox-locale-es17. firefox-locale-et18. firefox-locale-eu19. firefox-locale-fi20. firefox-locale-fr21. firefox-locale-fy22. firefox-locale-ga23. firefox-locale-gd24. firefox-locale-he25. firefox-locale-hr26. firefox-locale-id

125

Chapter D. Backends Specifications

27. firefox-locale-it28. firefox-locale-ko29. firefox-locale-lv30. firefox-locale-mk31. firefox-locale-ml32. firefox-locale-mn33. firefox-locale-mr34. firefox-locale-ms35. firefox-locale-nn36. firefox-locale-nso37. firefox-locale-oc38. firefox-locale-or39. firefox-locale-pa40. firefox-locale-pl41. firefox-locale-ro42. firefox-locale-si43. firefox-locale-sk44. firefox-locale-sl45. firefox-locale-sq46. firefox-locale-sr47. firefox-locale-sv48. firefox-locale-sw49. firefox-locale-ta50. firefox-locale-te51. firefox-locale-th52. firefox-locale-uk53. firefox-locale-vi54. firefox-locale-xh55. firefox-locale-zh-hans56. firefox-locale-zh-hant57. firefox-locale-zu58. gimp59. gimp-data60. gitstats61. gnuplot-nox62. groff63. heirloom-mailx64. imagemagick65. jadetex66. jq67. language-pack-af68. language-pack-af-base69. language-pack-ar70. language-pack-ar-base71. language-pack-ast

72. language-pack-ast-base73. language-pack-bg74. language-pack-bg-base75. language-pack-ca76. language-pack-ca-base77. language-pack-crh78. language-pack-crh-base79. language-pack-cy80. language-pack-cy-base81. language-pack-da82. language-pack-da-base83. language-pack-de84. language-pack-de-base85. language-pack-dv86. language-pack-dv-base87. language-pack-el88. language-pack-el-base89. language-pack-et90. language-pack-et-base91. language-pack-eu92. language-pack-eu-base93. language-pack-fi94. language-pack-fi-base95. language-pack-fil96. language-pack-fil-base97. language-pack-fy98. language-pack-fy-base99. language-pack-ga100. language-pack-ga-base101. language-pack-gd102. language-pack-gd-base103. language-pack-gnome-af104. language-pack-gnome-af-base105. language-pack-gnome-ar106. language-pack-gnome-ar-base107. language-pack-gnome-ast108. language-pack-gnome-ast-base109. language-pack-gnome-crh110. language-pack-gnome-crh-base111. language-pack-gnome-cy112. language-pack-gnome-cy-base113. language-pack-gnome-de114. language-pack-gnome-de-base115. language-pack-gnome-dv116. language-pack-gnome-dv-base

126

D.5 Packages installed on backend 1 and not 2

117. language-pack-gnome-et118. language-pack-gnome-et-base119. language-pack-gnome-fi120. language-pack-gnome-fi-base121. language-pack-gnome-fil122. language-pack-gnome-fil-base123. language-pack-gnome-fy124. language-pack-gnome-fy-base125. language-pack-gnome-ga126. language-pack-gnome-ga-base127. language-pack-gnome-gd128. language-pack-gnome-gd-base129. language-pack-gnome-he130. language-pack-gnome-he-base131. language-pack-gnome-hr132. language-pack-gnome-hr-base133. language-pack-gnome-ja134. language-pack-gnome-ja-base135. language-pack-gnome-ml136. language-pack-gnome-ml-base137. language-pack-gnome-ms138. language-pack-gnome-ms-base139. language-pack-gnome-nds140. language-pack-gnome-nds-base141. language-pack-gnome-ne142. language-pack-gnome-ne-base143. language-pack-gnome-nn144. language-pack-gnome-nn-base145. language-pack-gnome-sk146. language-pack-gnome-sk-base147. language-pack-gnome-so148. language-pack-gnome-so-base149. language-pack-gnome-th150. language-pack-gnome-th-base151. language-pack-he152. language-pack-he-base153. language-pack-hne154. language-pack-hne-base155. language-pack-hr156. language-pack-hr-base157. language-pack-hsb158. language-pack-hsb-base159. language-pack-hu160. language-pack-hu-base161. language-pack-id

162. language-pack-id-base163. language-pack-it164. language-pack-it-base165. language-pack-ja166. language-pack-ja-base167. language-pack-ko168. language-pack-ko-base169. language-pack-mhr170. language-pack-mhr-base171. language-pack-mi172. language-pack-mi-base173. language-pack-mk174. language-pack-mk-base175. language-pack-ml176. language-pack-ml-base177. language-pack-mn178. language-pack-mn-base179. language-pack-mr180. language-pack-mr-base181. language-pack-ms182. language-pack-ms-base183. language-pack-mt184. language-pack-mt-base185. language-pack-my186. language-pack-my-base187. language-pack-nan188. language-pack-nan-base189. language-pack-nds190. language-pack-nds-base191. language-pack-ne192. language-pack-ne-base193. language-pack-nl194. language-pack-nl-base195. language-pack-nn196. language-pack-nn-base197. language-pack-nso198. language-pack-nso-base199. language-pack-oc200. language-pack-oc-base201. language-pack-om202. language-pack-om-base203. language-pack-or204. language-pack-or-base205. language-pack-os206. language-pack-os-base

127

Chapter D. Backends Specifications

207. language-pack-pa208. language-pack-pa-base209. language-pack-pap210. language-pack-pap-base211. language-pack-pl212. language-pack-pl-base213. language-pack-ps214. language-pack-ps-base215. language-pack-ro216. language-pack-ro-base217. language-pack-rw218. language-pack-rw-base219. language-pack-sc220. language-pack-sc-base221. language-pack-sd222. language-pack-sd-base223. language-pack-se224. language-pack-se-base225. language-pack-shs226. language-pack-shs-base227. language-pack-si228. language-pack-si-base229. language-pack-sk230. language-pack-sk-base231. language-pack-sl232. language-pack-sl-base233. language-pack-so234. language-pack-so-base235. language-pack-sq236. language-pack-sq-base237. language-pack-sr238. language-pack-sr-base239. language-pack-ss240. language-pack-ss-base241. language-pack-st242. language-pack-st-base243. language-pack-sv244. language-pack-sv-base245. language-pack-sw246. language-pack-sw-base247. language-pack-ta248. language-pack-ta-base249. language-pack-te250. language-pack-te-base251. language-pack-tg

252. language-pack-tg-base253. language-pack-th254. language-pack-th-base255. language-pack-ti256. language-pack-ti-base257. language-pack-tk258. language-pack-tk-base259. language-pack-tl260. language-pack-tl-base261. language-pack-ts262. language-pack-ts-base263. language-pack-tt264. language-pack-tt-base265. language-pack-ug266. language-pack-ug-base267. language-pack-ur268. language-pack-ur-base269. language-pack-uz270. language-pack-uz-base271. language-pack-ve272. language-pack-ve-base273. language-pack-vi274. language-pack-vi-base275. language-pack-wa276. language-pack-wa-base277. language-pack-wae278. language-pack-wae-base279. language-pack-wo280. language-pack-wo-base281. language-pack-xh282. language-pack-xh-base283. language-pack-yi284. language-pack-yi-base285. language-pack-yo286. language-pack-yo-base287. language-pack-zh-hant288. language-pack-zh-hant-base289. language-pack-zu290. language-pack-zu-base291. libbabl-0.1-0:armhf292. libboost-iostreams1.54.0:armhf293. libcwidget3294. libgegl-0.2-0:armhf295. libgimp2.0296. libjavascriptcoregtk-1.0-0:armhf

128

D.6 Packages installed on backend 1 and not 3

297. liblua5.1-0:armhf298. libmng2:armhf299. libwebkitgtk-1.0-0:armhf300. libwebkitgtk-1.0-common301. locate302. ocl-icd-libopencl1:armhf303. opencl-headers304. psutils305. python-dateutil306. python-matplotlib307. python-matplotlib-data308. python-pyparsing

309. python-tk310. python-tz311. texlive-base312. texlive-fonts-recommended313. texlive-generic-recommended314. texlive-latex-base315. texlive-latex-recommended316. tipa317. wdiff318. wkhtmltopdf319. xmail

D.6 Packages installed on backend 1 and not 31. abiword2. abiword-common3. abiword-plugin-grammar4. abiword-plugin-mathview5. aglfn6. anthy7. anthy-common8. app-install-data9. aptitude

10. aptitude-common11. apturl12. apturl-common13. apt-xapian-index14. arduino15. arduino-core16. audacious17. audacious-plugins:armhf18. audacious-plugins-data19. autoconf2.1320. avahi-dnsconfd21. avrdude22. avr-libc23. binutils-avr24. bison25. bluez-alsa:armhf26. ca-certificates-java27. ccache28. cdbs29. checkinstall

30. chromium-browser31. chromium-browser-l10n32. chromium-codecs-ffmpeg-extra33. clinfo34. cmake35. cmake-data36. comerr-dev37. command-not-found-data38. consolekit39. cpp-4.840. culmus41. cups-driver-gutenprint42. deadbeef43. default-jre44. default-jre-headless45. dh-apparmor46. dh-translations47. docbook48. docbook-dsssl49. docbook-to-man50. docbook-utils51. docbook-xml52. docbook-xsl53. evince54. evince-common55. extra-xdg-menus56. faenza-icon-theme57. fakeroot58. file-roller

129

Chapter D. Backends Specifications

59. filezilla60. filezilla-common61. firefox62. firefox-locale-af63. firefox-locale-ar64. firefox-locale-bg65. firefox-locale-ca66. firefox-locale-cy67. firefox-locale-da68. firefox-locale-de69. firefox-locale-en70. firefox-locale-es71. firefox-locale-et72. firefox-locale-eu73. firefox-locale-fi74. firefox-locale-fr75. firefox-locale-fy76. firefox-locale-ga77. firefox-locale-gd78. firefox-locale-he79. firefox-locale-hr80. firefox-locale-id81. firefox-locale-it82. firefox-locale-ko83. firefox-locale-lv84. firefox-locale-mk85. firefox-locale-ml86. firefox-locale-mn87. firefox-locale-mr88. firefox-locale-ms89. firefox-locale-nn90. firefox-locale-nso91. firefox-locale-oc92. firefox-locale-or93. firefox-locale-pa94. firefox-locale-pl95. firefox-locale-ro96. firefox-locale-si97. firefox-locale-sk98. firefox-locale-sl99. firefox-locale-sq100. firefox-locale-sr101. firefox-locale-sv102. firefox-locale-sw103. firefox-locale-ta

104. firefox-locale-te105. firefox-locale-th106. firefox-locale-uk107. firefox-locale-vi108. firefox-locale-xh109. firefox-locale-zh-hans110. firefox-locale-zh-hant111. firefox-locale-zu112. flex113. flite1-dev:armhf114. fonts-arabeyes115. fonts-arphic-ukai116. fonts-arphic-uming117. fonts-dejavu118. fonts-dejavu-extra119. fonts-droid120. fonts-farsiweb121. fonts-khmeros122. fonts-liberation123. fonts-lyx124. fonts-manchufont125. fonts-mgopen126. fonts-nafees127. fonts-nanum128. fonts-nanum-coding129. fonts-sil-ezra130. fonts-sil-scheherazade131. fonts-takao-gothic132. fonts-takao-mincho133. fonts-ukij-uyghur134. fonts-unfonts-core135. g++-4.8136. gcc-4.8137. gcc-4.8-base:armhf138. gcc-6-base:armhf139. gcc-avr140. gdebi141. gdebi-core142. gecko-mediaplayer143. gfortran-4.8144. giblib1:armhf145. gimp146. gimp-data147. gir1.2-clutter-1.0148. gir1.2-clutter-gst-2.0

130

D.6 Packages installed on backend 1 and not 3

149. gir1.2-cogl-1.0150. gir1.2-coglpango-1.0151. gir1.2-freedesktop152. gir1.2-gconf-2.0153. gir1.2-gdkpixbuf-2.0154. gir1.2-glib-2.0155. gir1.2-gtk-3.0156. gir1.2-gtkclutter-1.0157. gir1.2-gudev-1.0158. gir1.2-ibus-1.0159. gir1.2-javascriptcoregtk-3.0160. gir1.2-json-1.0161. gir1.2-pango-1.0162. gir1.2-polkit-1.0163. gir1.2-rsvg-2.0164. gir1.2-vte-2.90165. gir1.2-webkit-3.0166. gir1.2-wnck-3.0167. gitstats168. gnome-common169. gnome-desktop-data170. gnome-disk-utility171. gnome-icon-theme-full172. gnome-mplayer173. gnome-panel174. gnome-panel-data175. gnome-pkg-tools176. gnome-system-monitor177. gnome-system-tools178. gnumeric179. gnumeric-common180. gnumeric-doc181. gnuplot-nox182. gobject-introspection183. gparted184. gperf185. gpicview186. groff187. gsfonts-x11188. gstreamer0.10-nice:armhf189. gstreamer0.10-plugins-base:armhf190. gstreamer0.10-plugins-good:armhf191. gstreamer0.10-x:armhf192. gstreamer1.0-alsa:armhf193. gstreamer1.0-doc

194. gstreamer1.0-libav:armhf195. gstreamer1.0-plugins-bad:armhf196. gstreamer1.0-plugins-bad-doc197. gstreamer1.0-plugins-base-apps198. gstreamer1.0-plugins-base-doc199. gstreamer1.0-plugins-good-doc200. gstreamer1.0-tools201. gtk3-engines-unico:armhf202. gtk-doc-tools203. gtk-im-libthai:armhf204. gucharmap205. gvfs-fuse206. hardening-wrapper207. hardinfo208. heirloom-mailx209. hunspell-ar210. hunspell-be211. hunspell-da212. hunspell-de-at213. hunspell-de-ch214. hunspell-de-de215. hunspell-eu-es216. hunspell-fr217. hunspell-fr-classical218. hunspell-gl-es219. hunspell-hu220. hunspell-ko221. hunspell-ml222. hunspell-ne223. hunspell-ro224. hunspell-ru225. hunspell-sr226. hunspell-sv-se227. hunspell-uz228. hunspell-vi229. hyphen-af230. hyphen-as231. hyphen-bn232. hyphen-ca233. hyphen-de234. hyphen-fr235. hyphen-gu236. hyphen-hi237. hyphen-hr238. hyphen-hu

131

Chapter D. Backends Specifications

239. hyphen-it240. hyphen-kn241. hyphen-mr242. hyphen-pa243. hyphen-pl244. hyphen-ro245. hyphen-ru246. hyphen-sl247. hyphen-sr248. hyphen-ta249. hyphen-te250. hyphen-zu251. ibus-anthy252. ibus-chewing253. ibus-hangul254. ibus-m17n255. ibus-sunpinyin256. ibus-table257. ibus-table-cangjie3258. ibus-table-cangjie5259. ibus-table-quick-classic260. ibus-table-wubi261. ibus-unikey262. icedtea-7-jre-jamvm:armhf263. indicator-application-gtk2264. intltool265. iotop266. jade267. jadetex268. java-common269. jq270. krb5-multidev271. ladspa-sdk272. language-pack-af273. language-pack-af-base274. language-pack-ar275. language-pack-ar-base276. language-pack-ast277. language-pack-ast-base278. language-pack-bg279. language-pack-bg-base280. language-pack-ca281. language-pack-ca-base282. language-pack-crh283. language-pack-crh-base

284. language-pack-cy285. language-pack-cy-base286. language-pack-da287. language-pack-da-base288. language-pack-de289. language-pack-de-base290. language-pack-dv291. language-pack-dv-base292. language-pack-el293. language-pack-el-base294. language-pack-et295. language-pack-et-base296. language-pack-eu297. language-pack-eu-base298. language-pack-fi299. language-pack-fi-base300. language-pack-fil301. language-pack-fil-base302. language-pack-fy303. language-pack-fy-base304. language-pack-ga305. language-pack-ga-base306. language-pack-gd307. language-pack-gd-base308. language-pack-gnome-af309. language-pack-gnome-af-base310. language-pack-gnome-ar311. language-pack-gnome-ar-base312. language-pack-gnome-ast313. language-pack-gnome-ast-base314. language-pack-gnome-crh315. language-pack-gnome-crh-base316. language-pack-gnome-cy317. language-pack-gnome-cy-base318. language-pack-gnome-de319. language-pack-gnome-de-base320. language-pack-gnome-dv321. language-pack-gnome-dv-base322. language-pack-gnome-et323. language-pack-gnome-et-base324. language-pack-gnome-fi325. language-pack-gnome-fi-base326. language-pack-gnome-fil327. language-pack-gnome-fil-base328. language-pack-gnome-fy

132

D.6 Packages installed on backend 1 and not 3

329. language-pack-gnome-fy-base330. language-pack-gnome-ga331. language-pack-gnome-ga-base332. language-pack-gnome-gd333. language-pack-gnome-gd-base334. language-pack-gnome-he335. language-pack-gnome-he-base336. language-pack-gnome-hr337. language-pack-gnome-hr-base338. language-pack-gnome-ja339. language-pack-gnome-ja-base340. language-pack-gnome-ml341. language-pack-gnome-ml-base342. language-pack-gnome-ms343. language-pack-gnome-ms-base344. language-pack-gnome-nds345. language-pack-gnome-nds-base346. language-pack-gnome-ne347. language-pack-gnome-ne-base348. language-pack-gnome-nn349. language-pack-gnome-nn-base350. language-pack-gnome-sk351. language-pack-gnome-sk-base352. language-pack-gnome-so353. language-pack-gnome-so-base354. language-pack-gnome-th355. language-pack-gnome-th-base356. language-pack-he357. language-pack-he-base358. language-pack-hne359. language-pack-hne-base360. language-pack-hr361. language-pack-hr-base362. language-pack-hsb363. language-pack-hsb-base364. language-pack-hu365. language-pack-hu-base366. language-pack-id367. language-pack-id-base368. language-pack-it369. language-pack-it-base370. language-pack-ja371. language-pack-ja-base372. language-pack-ko373. language-pack-ko-base

374. language-pack-mhr375. language-pack-mhr-base376. language-pack-mi377. language-pack-mi-base378. language-pack-mk379. language-pack-mk-base380. language-pack-ml381. language-pack-ml-base382. language-pack-mn383. language-pack-mn-base384. language-pack-mr385. language-pack-mr-base386. language-pack-ms387. language-pack-ms-base388. language-pack-mt389. language-pack-mt-base390. language-pack-my391. language-pack-my-base392. language-pack-nan393. language-pack-nan-base394. language-pack-nb395. language-pack-nb-base396. language-pack-nds397. language-pack-nds-base398. language-pack-ne399. language-pack-ne-base400. language-pack-nl401. language-pack-nl-base402. language-pack-nn403. language-pack-nn-base404. language-pack-nso405. language-pack-nso-base406. language-pack-oc407. language-pack-oc-base408. language-pack-om409. language-pack-om-base410. language-pack-or411. language-pack-or-base412. language-pack-os413. language-pack-os-base414. language-pack-pa415. language-pack-pa-base416. language-pack-pap417. language-pack-pap-base418. language-pack-pl

133

Chapter D. Backends Specifications

419. language-pack-pl-base420. language-pack-ps421. language-pack-ps-base422. language-pack-ro423. language-pack-ro-base424. language-pack-rw425. language-pack-rw-base426. language-pack-sc427. language-pack-sc-base428. language-pack-sd429. language-pack-sd-base430. language-pack-se431. language-pack-se-base432. language-pack-shs433. language-pack-shs-base434. language-pack-si435. language-pack-si-base436. language-pack-sk437. language-pack-sk-base438. language-pack-sl439. language-pack-sl-base440. language-pack-so441. language-pack-so-base442. language-pack-sq443. language-pack-sq-base444. language-pack-sr445. language-pack-sr-base446. language-pack-ss447. language-pack-ss-base448. language-pack-st449. language-pack-st-base450. language-pack-sv451. language-pack-sv-base452. language-pack-sw453. language-pack-sw-base454. language-pack-ta455. language-pack-ta-base456. language-pack-te457. language-pack-te-base458. language-pack-tg459. language-pack-tg-base460. language-pack-th461. language-pack-th-base462. language-pack-ti463. language-pack-ti-base

464. language-pack-tk465. language-pack-tk-base466. language-pack-tl467. language-pack-tl-base468. language-pack-ts469. language-pack-ts-base470. language-pack-tt471. language-pack-tt-base472. language-pack-ug473. language-pack-ug-base474. language-pack-ur475. language-pack-ur-base476. language-pack-uz477. language-pack-uz-base478. language-pack-ve479. language-pack-ve-base480. language-pack-vi481. language-pack-vi-base482. language-pack-wa483. language-pack-wa-base484. language-pack-wae485. language-pack-wae-base486. language-pack-wo487. language-pack-wo-base488. language-pack-xh489. language-pack-xh-base490. language-pack-yi491. language-pack-yi-base492. language-pack-yo493. language-pack-yo-base494. language-pack-zh-hant495. language-pack-zh-hant-base496. language-pack-zu497. language-pack-zu-base498. leafpad499. liba52-0.7.4500. libaa1-dev501. libabiword-3.0:armhf502. libamd2.3.1:armhf503. libanthy0:armhf504. libapt-inst1.5:armhf505. libapt-pkg4.12:armhf506. libart-2.0-dev507. libasan0:armhf508. libasound2-dev:armhf

134

D.6 Packages installed on backend 1 and not 3

509. libaspell15510. libasprintf0c2:armhf511. libass4:armhf512. libass-dev:armhf513. libatk1.0-dev514. libatk-bridge2.0-dev:armhf515. libatkmm-1.6-1:armhf516. libatk-wrapper-java517. libatk-wrapper-java-jni:armhf518. libatomic-ops-dev519. libaudclient2:armhf520. libaudcore1:armhf521. libaudit-dev522. libautodie-perl523. libavahi-client-dev524. libavahi-common-dev525. libavahi-glib-dev526. libavc1394-dev:armhf527. libavcodec54:armhf528. libavcodec-dev529. libavformat54:armhf530. libavformat-dev531. libavresample1:armhf532. libavutil52:armhf533. libavutil-dev534. libbabl-0.1-0:armhf535. libbinio1ldbl:armhf536. libbison-dev:armhf537. libbluetooth-dev538. libbonobo2-0:armhf539. libbonobo2-common540. libbonobo2-dev:armhf541. libbonoboui2-0:armhf542. libbonoboui2-common543. libbonoboui2-dev:armhf544. libboost1.54-dev545. libboost-atomic1.54.0:armhf546. libboost-atomic1.54-dev:armhf547. libboost-chrono1.54.0:armhf548. libboost-chrono1.54-dev:armhf549. libboost-date-time1.54.0:armhf550. libboost-date-time1.54-dev:armhf551. libboost-dev552. libboost-iostreams1.54.0:armhf553. libboost-serialization1.54.0:armhf

554. libboost-serialization1.54-dev:armhf555. libboost-system1.54.0:armhf556. libboost-system1.54-dev:armhf557. libboost-thread1.54.0:armhf558. libboost-thread1.54-dev:armhf559. libboost-thread-dev:armhf560. libbs2b0561. libbz2-dev:armhf562. libcaca-dev563. libcairo2-dev564. libcairomm-1.0-1:armhf565. libcairo-perl566. libcairo-script-interpreter2:armhf567. libcamd2.3.1:armhf568. libcamel-1.2-45569. libcanberra-dev:armhf570. libcap-dev:armhf571. libcap-ng0572. libccolamd2.8.0:armhf573. libcdaudio1574. libcdaudio-dev575. libcddb2-dev576. libcdio-dev577. libcdparanoia-dev:armhf578. libcdt5579. libcgraph6580. libchamplain-0.12-0:armhf581. libchamplain-gtk-0.12-0:armhf582. libchewing3:armhf583. libchewing3-data:armhf584. libcholmod2.1.2:armhf585. libchromaprint-dev586. libclutter-1.0-dev587. libclutter-gst-2.0-dev588. libclutter-gtk-1.0-dev589. libcogl15:armhf590. libcogl-dev591. libcogl-pango15:armhf592. libcogl-pango-dev593. libcolord1:armhf594. libcolorhug1:armhf595. libcompfaceg1596. libcue1597. libcurl4-gnutls-dev:armhf598. libcvaux-dev:armhf

135

Chapter D. Backends Specifications

599. libcv-dev:armhf600. libcwidget3601. libdaemon0602. libdbus-1-dev:armhf603. libdbus-glib-1-dev604. libdc1394-22-dev:armhf605. libdca-dev:armhf606. libdirac-decoder0:armhf607. libdirac-dev:armhf608. libdirac-encoder0:armhf609. libdirectfb-dev610. libdirectfb-extra:armhf611. libdiscid0:armhf612. libdjvulibre-dev:armhf613. libdmx1:armhf614. libdmx-dev:armhf615. libdv4-dev:armhf616. libdvdnav-dev:armhf617. libdvdread-dev:armhf618. libebook-contacts-1.2-0619. libecal-1.2-16620. libedataserver-1.2-18621. libegl1-mesa-dev622. libegl1-mesa-drivers:armhf623. libelfg0:armhf624. libenca-dev625. libenchant-voikko:armhf626. libept1.4.12:armhf627. libevdocument3-4628. libevview3-3629. libexempi-dev:armhf630. libexif-dev631. libexo-1-0:armhf632. libexo-common633. libexo-helpers634. libfaad-dev:armhf635. libfakeroot:armhf636. libfarstream-0.1-0:armhf637. libffi-dev:armhf638. libffmpegthumbnailer4639. libfftw3-bin640. libfftw3-dev:armhf641. libflac-dev:armhf642. libfl-dev:armhf643. libflite1:armhf

644. libfluidsynth1:armhf645. libfluidsynth-dev:armhf646. libfm4647. libfm-data648. libfm-extra4649. libfm-gtk4650. libfm-gtk-data651. libfm-modules652. libfontconfig1-dev653. libfontenc-dev:armhf654. libframe6:armhf655. libfreetype6-dev656. libfribidi-dev657. libfs6:armhf658. libftdi1:armhf659. libgail-3-0:armhf660. libgail-common:armhf661. libgail-dev662. libgbm-dev663. libgcc-4.8-dev:armhf664. libgconf2-4:armhf665. libgconf2-dev666. libgcrypt11:armhf667. libgcrypt11-dev668. libgda-5.0-4669. libgda-5.0-common670. libgdk-pixbuf2.0-dev671. libgdome2-0672. libgdome2-cpp-smart0c2a673. libgegl-0.2-0:armhf674. libgeis1:armhf675. libgfortran-4.8-dev:armhf676. libgif-dev677. libgimp2.0678. libgirepository-1.0-1679. libgirepository1.0-dev680. libgl1-mesa-dev681. libglade2-0:armhf682. libgles1-mesa-dev683. libgles2-mesa-dev684. libglib2.0-dev685. libglib2.0-doc686. libglibmm-2.4-1c2a:armhf687. libglib-perl688. libglu1-mesa-dev

136

D.6 Packages installed on backend 1 and not 3

689. libgme0690. libgme-dev691. libgmlib1:armhf692. libgmp3-dev693. libgmp-dev:armhf694. libgmpxx4ldbl:armhf695. libgmtk1:armhf696. libgmtk1-data697. libgnome2-0:armhf698. libgnome2-bin699. libgnome2-common700. libgnome2-dev:armhf701. libgnome-bluetooth11702. libgnomecanvas2-0:armhf703. libgnomecanvas2-common704. libgnomecanvas2-dev:armhf705. libgnome-desktop-3-7706. libgnome-keyring-dev707. libgnomeui-0:armhf708. libgnomeui-common709. libgnomeui-dev:armhf710. libgnomevfs2-0:armhf711. libgnomevfs2-common712. libgnomevfs2-dev:armhf713. libgnutls26:armhf714. libgnutls-dev715. libgnutlsxx27:armhf716. libgoffice-0.10-10717. libgoffice-0.10-10-common718. libgpg-error-dev719. libgphoto2-port10:armhf720. libgrail6721. libgraphviz-dev722. libgrip0723. libgsf-1-114724. libgsf-1-common725. libgsl0-dev726. libgsl0ldbl727. libgsm1-dev:armhf728. libgssrpc4:armhf729. libgstreamer0.10-0:armhf730. libgstreamer1.0-dev731. libgstreamer-plugins-bad1.0-0:armhf732. libgstreamer-plugins-bad1.0-dev733. libgstreamer-plugins-base0.10-

0:armhf734. libgstreamer-plugins-base1.0-dev735. libgstreamer-plugins-good1.0-dev736. libgtk2.0-dev737. libgtk2-perl738. libgtk-3-dev739. libgtkmathview0c2a740. libgtkmm-2.4-1c2a:armhf741. libgtkmm-3.0-1:armhf742. libgtkspell0743. libgtop2-7744. libgucharmap-2-90-7745. libgudev-1.0-dev746. libguess1:armhf747. libgupnp-igd-1.0-4:armhf748. libgvc6749. libgvpr2750. libgweather-3-6751. libhangul1:armhf752. libhangul-data753. libharfbuzz-dev754. libharfbuzz-gobject0:armhf755. libhighgui-dev:armhf756. libhogweed2:armhf757. libical1758. libice-dev:armhf759. libicu52:armhf760. libid3tag0761. libid3tag0-dev762. libidl0:armhf763. libidl-common764. libidl-dev:armhf765. libidn11-dev766. libiec61883-dev767. libijs-0.35768. libilmbase6:armhf769. libilmbase-dev770. libimage-exiftool-perl771. libimlib2772. libimlib2-dev773. libiptcdata0774. libiptcdata0-dev775. libisl15:armhf776. libiso9660-dev777. libiw-dev:armhf

137

Chapter D. Backends Specifications

778. libjack-jackd2-dev:armhf779. libjasper-dev780. libjbig-dev:armhf781. libjna-java782. libjpeg8-dev:armhf783. libjpeg-dev:armhf784. libjpeg-progs785. libjpeg-turbo8-dev:armhf786. libjpeg-turbo-progs787. libjson0:armhf788. libjson-glib-dev789. libkadm5clnt-mit9:armhf790. libkadm5srv-mit9:armhf791. libkate-dev792. libkdb5-7:armhf793. libkrb5-dev794. liblavfile-2.1-0795. liblavjpeg-2.1-0796. liblavplay-2.1-0797. liblcms2-dev:armhf798. libldap2-dev:armhf799. liblink-grammar4800. libllvm3.4:armhf801. liblockfile1:armhf802. liblockfile-bin803. libloudmouth1-0804. liblqr-1-0-dev805. libltdl-dev:armhf806. liblzma-dev:armhf807. liblzo2-dev:armhf808. libm17n-0809. libmad0-dev810. libmagick++5:armhf811. libmagickcore5:armhf812. libmagickcore5-extra:armhf813. libmagickcore-dev814. libmagick++-dev815. libmagickwand5:armhf816. libmagickwand-dev817. libmbim-glib0:armhf818. libmeanwhile1819. libmenu-cache3820. libmenu-cache-bin821. libmessaging-menu0822. libmetacity-private0a

823. libmicrohttpd-dev824. libmikmod2:armhf825. libmikmod2-dev:armhf826. libmimic0827. libmimic-dev828. libminiupnpc8829. libmirclient7:armhf830. libmirclientplatform-mesa:armhf831. libmirprotobuf0:armhf832. libmirprotobuf-dev:armhf833. libmjpegtools-dev834. libmjpegutils-2.1-0835. libmms0:armhf836. libmms-dev:armhf837. libmodplug1838. libmodplug-dev839. libmowgli2:armhf840. libmp3lame-dev:armhf841. libmpcdec6842. libmpcdec-dev843. libmpeg2-4-dev:armhf844. libmpeg2encpp-2.1-0845. libmpeg3-1846. libmpeg3-dev847. libmpg123-0:armhf848. libmpg123-dev:armhf849. libmplex2-2.1-0850. libmusicbrainz3-6851. libmysqlclient-dev852. libncurses5-dev:armhf853. libnettle4:armhf854. libnfs1:armhf855. libnfs-dev:armhf856. libnice10:armhf857. libnm-glib4858. libnm-glib-vpn1859. libnm-gtk0860. libnm-util2861. libnotify-bin862. libnotify-dev863. libnss3-1d:armhf864. libntdb1:armhf865. libobrender29866. libobt2867. libofa0

138

D.6 Packages installed on backend 1 and not 3

868. libofa0-dev869. libogg-dev:armhf870. libonig2871. liboobs-1-5872. libopenal1:armhf873. libopenal-data874. libopenal-dev:armhf875. libopencv2.4-java876. libopencv2.4-jni877. libopencv-calib3d2.4:armhf878. libopencv-calib3d-dev:armhf879. libopencv-contrib2.4:armhf880. libopencv-contrib-dev:armhf881. libopencv-core2.4:armhf882. libopencv-core-dev:armhf883. libopencv-dev884. libopencv-features2d2.4:armhf885. libopencv-features2d-dev:armhf886. libopencv-flann2.4:armhf887. libopencv-flann-dev:armhf888. libopencv-gpu2.4:armhf889. libopencv-gpu-dev:armhf890. libopencv-highgui2.4:armhf891. libopencv-highgui-dev:armhf892. libopencv-imgproc2.4:armhf893. libopencv-imgproc-dev:armhf894. libopencv-legacy2.4:armhf895. libopencv-legacy-dev:armhf896. libopencv-ml2.4:armhf897. libopencv-ml-dev:armhf898. libopencv-objdetect2.4:armhf899. libopencv-objdetect-dev:armhf900. libopencv-ocl2.4:armhf901. libopencv-ocl-dev:armhf902. libopencv-photo2.4:armhf903. libopencv-photo-dev:armhf904. libopencv-stitching2.4:armhf905. libopencv-stitching-dev:armhf906. libopencv-superres2.4:armhf907. libopencv-superres-dev:armhf908. libopencv-ts2.4:armhf909. libopencv-ts-dev:armhf910. libopencv-video2.4:armhf911. libopencv-video-dev:armhf912. libopencv-videostab2.4:armhf

913. libopencv-videostab-dev:armhf914. libopenexr6:armhf915. libopenexr-dev916. libopenjpeg2:armhf917. libopenjpeg-dev918. libopenvg1-mesa:armhf919. libopts25:armhf920. libopus0921. libopus-dev922. liborbit-2-0:armhf923. liborbit2:armhf924. liborbit2-dev925. liborc-0.4-dev926. libotf0:armhf927. libots0928. libp11-kit-dev929. libpam-cap:armhf930. libpanel-applet-4-0931. libpango1.0-dev932. libpangomm-1.4-1:armhf933. libpango-perl934. libparted0debian1:armhf935. libpathplan4936. libpcre3-dev:armhf937. libpcrecpp0:armhf938. libperl5.18939. libpisock9940. libplist1:armhf941. libplist-dev942. libplymouth2:armhf943. libpng12-dev944. libpolkit-agent-1-dev945. libpolkit-gobject-1-dev946. libpoppler44:armhf947. libpopt-dev:armhf948. libpostproc52949. libprotobuf8:armhf950. libprotobuf-lite8:armhf951. libproxy1:armhf952. libpulse-dev:armhf953. libpurple0954. libqmi-glib0:armhf955. libqpdf13:armhf956. libquicktime2:armhf957. librarian0

139

Chapter D. Backends Specifications

958. libraw1394-dev:armhf959. libreadline5:armhf960. libreadline6-dev:armhf961. libreadline-dev:armhf962. librsvg2-dev963. librtmp0:armhf964. librtmp-dev965. librxtx-java966. libsamplerate0-dev:armhf967. libsbc1:armhf968. libsbc-dev:armhf969. libschroedinger-dev:armhf970. libsdl1.2-dev971. libsdl-gfx1.2-4:armhf972. libsdl-gfx1.2-dev:armhf973. libsdl-image1.2-dev:armhf974. libsdl-mixer1.2:armhf975. libsdl-mixer1.2-dev:armhf976. libselinux1-dev:armhf977. libsepol1-dev978. libsgmls-perl979. libshout3-dev:armhf980. libsidplayfp:armhf981. libsigc++-2.0-0c2a:armhf982. libslang2-dev:armhf983. libsmbclient-dev:armhf984. libsm-dev:armhf985. libsndfile1-dev986. libsoundtouch0:armhf987. libsoundtouch-dev988. libsoup2.4-dev989. libsp1c2990. libspandsp2991. libspandsp-dev992. libspeex-dev:armhf993. libsqlite0994. libsqlite3-dev:armhf995. libsrtp0996. libsrtp0-dev997. libssh2-1-dev:armhf998. libssh-dev999. libssl-dev:armhf1000. libstartup-notification0-dev:armhf1001. libstdc++-4.8-dev:armhf1002. libsunpinyin3:armhf

1003. libswscale2:armhf1004. libswscale-dev1005. libsystemd-daemon0:armhf1006. libsystemd-login0:armhf1007. libt1-51008. libtag1c2a:armhf1009. libtag1-dev1010. libtag1-vanilla:armhf1011. libtagc0:armhf1012. libtagc0-dev1013. libtasn1-6-dev1014. libtelepathy-glib0:armhf1015. libtheora-dev:armhf1016. libtidy-0.99-01017. libtiff5-dev:armhf1018. libtiffxx5:armhf1019. libtinfo-dev:armhf1020. libtinyxml2.6.2:armhf1021. libtinyxml-dev:armhf1022. libts-0.0-0:armhf1023. libudev-dev1024. libumfpack5.6.2:armhf1025. libuniconf4.61026. libupower-glib1:armhf1027. libusb-1.0-0-dev:armhf1028. libusb-dev1029. libusbmuxd21030. libv4l2rds0:armhf1031. libv4l-dev:armhf1032. libvisual-0.4-dev1033. libvncserver0:armhf1034. libvo-aacenc0:armhf1035. libvo-aacenc-dev:armhf1036. libvo-amrwbenc0:armhf1037. libvo-amrwbenc-dev:armhf1038. libvoikko1:armhf1039. libvorbis-dev:armhf1040. libvpx1:armhf1041. libvpx-dev:armhf1042. libvte-2.90-91043. libvte-2.90-common1044. libwavpack-dev:armhf1045. libwebpdemux1:armhf1046. libwebp-dev:armhf1047. libwildmidi1:armhf

140

D.6 Packages installed on backend 1 and not 3

1048. libwildmidi-config1049. libwildmidi-dev1050. libwmf-dev1051. libwpd-0.9-91052. libwpg-0.2-21053. libwps-0.2-21054. libwv-1.2-4:armhf1055. libwvstreams4.6-base1056. libwvstreams4.6-extras1057. libwxbase2.8-0:armhf1058. libwxgtk2.8-0:armhf1059. libx264-142:armhf1060. libxapian221061. libxaw7-dev:armhf1062. libxcb-icccm4-dev:armhf1063. libxcb-image0-dev:armhf1064. libxcb-keysyms1-dev:armhf1065. libxcb-shm0-dev:armhf1066. libxcb-util0:armhf1067. libxcb-util0-dev:armhf1068. libxcb-xf86dri0:armhf1069. libxcb-xf86dri0-dev:armhf1070. libxcb-xv0-dev:armhf1071. libxcomposite-dev1072. libxcursor-dev:armhf1073. libxdot41074. libxfce4ui-1-01075. libxfce4ui-2-01076. libxfce4ui-2-dev1077. libxfce4ui-common1078. libxfce4util61079. libxfce4util-common1080. libxfce4util-dev1081. libxfconf-0-21082. libxfconf-0-dev1083. libxfont-dev1084. libxft-dev1085. libxi-dev1086. libxinerama-dev:armhf1087. libxml2-dev:armhf1088. libxml2-utils1089. libxmu-dev:armhf1090. libxmu-headers1091. libxmuu-dev:armhf1092. libxp6:armhf

1093. libxpm-dev:armhf1094. libxrandr-dev:armhf1095. libxrender-dev:armhf1096. libxres-dev1097. libxslt1-dev:armhf1098. libxt-dev:armhf1099. libxtst-dev:armhf1100. libxv-dev:armhf1101. libxvidcore-dev:armhf1102. libyajl-dev1103. libzbar01104. libzbar-dev1105. libzephyr4:armhf1106. libzip21107. libzip-dev1108. libzvbi-dev:armhf1109. light-locker1110. light-locker-settings1111. link-grammar-dictionaries-en1112. localepurge1113. locate1114. lockfile-progs1115. lsof1116. luatex1117. lubuntu-artwork1118. lubuntu-artwork-14-041119. lubuntu-icon-theme1120. lubuntu-lxpanel-icons1121. lubuntu-software-center1122. lxappearance1123. lxappearance-obconf1124. lxde-common1125. lxde-core1126. lxinput1127. lxlauncher1128. lxmenu-data1129. lxpanel1130. lxpanel-indicator-applet-plugin1131. lxrandr1132. lxsession1133. lxsession-data1134. lxsession-default-apps1135. lxsession-edit1136. lxsession-logout1137. lxshortcut

141

Chapter D. Backends Specifications

1138. lxtask1139. lxterminal1140. lynx1141. lynx-cur1142. m17n-contrib1143. m17n-db1144. mc1145. mc-data1146. medit1147. mesa-common-dev1148. metacity1149. metacity-common1150. minicom1151. mircommon-dev:armhf1152. mountall1153. mplayer21154. mtpaint1155. nano1156. nettle-dev1157. ntp1158. ntpdate1159. obconf1160. ocl-icd-libopencl1:armhf1161. openbox1162. opencl-headers1163. openjdk-7-jre:armhf1164. openjdk-7-jre-headless:armhf1165. oracle-java8-installer1166. orbit21167. pastebinit1168. pavucontrol1169. pcmanfm1170. perl-doc1171. pidgin1172. pidgin-data1173. pidgin-libnotify1174. plymouth1175. plymouth-label1176. plymouth-theme-lubuntu-logo1177. plymouth-theme-lubuntu-text1178. pm-utils1179. psutils1180. python-aptdaemon1181. python-aptdaemon.gtk3widgets1182. python-colorama

1183. python-commandnotfound1184. python-cups1185. python-cupshelpers1186. python-dateutil1187. python-debian1188. python-defer1189. python-distlib1190. python-gconf1191. python-gdbm1192. python-glade21193. python-gnomekeyring1194. python-gudev1195. python-html5lib1196. python-libxml21197. python-mako1198. python-markupsafe1199. python-matplotlib1200. python-matplotlib-data1201. python-notify1202. python-ntdb1203. python-pexpect1204. python-pil1205. python-pip1206. python-psutil1207. python-pycurl1208. python-pyinotify1209. python-pyparsing1210. python-pysqlite21211. python-renderpm1212. python-reportlab1213. python-reportlab-accel1214. python-scour1215. python-setuptools1216. python-smbc1217. python-sqlite1218. python-support1219. python-tk1220. python-tz1221. python-wheel1222. python-xapian1223. python-xdg1224. rarian-compat1225. realpath1226. scrot1227. sessioninstaller

142

D.7 Packages installed on backend 2 and not 1

1228. sgml-data1229. sgmlspl1230. simple-scan1231. smbclient1232. sp1233. sunpinyin-data1234. swig1235. swig2.01236. sylpheed1237. sylpheed-doc1238. sylpheed-i18n1239. sylpheed-plugins1240. synaptic1241. systemd-services1242. system-tools-backends1243. texlive-fonts-recommended1244. texlive-generic-recommended1245. texlive-latex-base1246. texlive-latex-recommended1247. tipa1248. transfig1249. transmission1250. tsconf1251. ttf-bengali-fonts1252. ttf-devanagari-fonts1253. ttf-gujarati-fonts1254. ttf-kannada-fonts1255. ttf-malayalam-fonts1256. ttf-oriya-fonts1257. ttf-punjabi-fonts1258. ttf-tamil-fonts1259. ttf-telugu-fonts1260. tzdata-java1261. ubuntu-extras-keyring1262. upstart1263. valgrind1264. voikko-fi1265. wbulgarian1266. wdiff1267. wfrench1268. wirish

1269. witalian1270. wkhtmltopdf1271. wmanx1272. wogerman1273. wspanish1274. wswedish1275. wvdial1276. x11proto-bigreqs-dev1277. x11proto-composite-dev1278. x11proto-dmx-dev1279. x11proto-record-dev1280. x11proto-xcmisc-dev1281. x11proto-xf86dga-dev1282. x11vnc1283. x11vnc-data1284. x11-xfs-utils1285. xarchiver1286. xdg-user-dirs-gtk1287. xfburn1288. xfce4-dev-tools1289. xfce4-notifyd1290. xfce4-power-manager1291. xfce4-power-manager-data1292. xfconf1293. xfonts-100dpi1294. xmail1295. xmlto1296. xpad1297. xscreensaver1298. xscreensaver-data1299. xscreensaver-data-extra1300. xscreensaver-screensaver-bsod1301. xserver-xorg-input-multitouch1302. xserver-xorg-video-modesetting1303. xserver-xorg-video-omap1304. xserver-xorg-video-vesa1305. xsltproc1306. xul-ext-mozvoikko1307. xul-ext-ubufox1308. xvfb1309. yasm

D.7 Packages installed on backend 2 and not 1

143

Chapter D. Backends Specifications

1. aspell2. aspell-en3. firefox-locale-nb4. language-pack-gnome-nb

5. language-pack-gnome-nb-base6. libisl10:armhf7. wnorwegian

D.8 Packages installed on backend 2 and not 31. abiword2. abiword-common3. abiword-plugin-grammar4. abiword-plugin-mathview5. anthy6. anthy-common7. app-install-data8. apturl9. apturl-common

10. arduino11. arduino-core12. audacious13. audacious-plugins:armhf14. audacious-plugins-data15. autoconf2.1316. avahi-dnsconfd17. avrdude18. avr-libc19. binutils-avr20. bison21. bluez-alsa:armhf22. ca-certificates-java23. ccache24. cdbs25. checkinstall26. chromium-codecs-ffmpeg-extra27. cmake28. cmake-data29. comerr-dev30. command-not-found-data31. consolekit32. cpp-4.833. culmus34. cups-driver-gutenprint35. deadbeef36. default-jre37. default-jre-headless

38. dh-apparmor39. dh-translations40. docbook41. docbook-dsssl42. docbook-to-man43. docbook-xml44. docbook-xsl45. evince46. evince-common47. extra-xdg-menus48. faenza-icon-theme49. fakeroot50. file-roller51. filezilla52. filezilla-common53. firefox54. firefox-locale-en55. firefox-locale-nb56. flex57. flite1-dev:armhf58. fonts-arabeyes59. fonts-arphic-ukai60. fonts-arphic-uming61. fonts-dejavu62. fonts-dejavu-extra63. fonts-droid64. fonts-farsiweb65. fonts-khmeros66. fonts-liberation67. fonts-lyx68. fonts-manchufont69. fonts-mgopen70. fonts-nafees71. fonts-nanum72. fonts-nanum-coding73. fonts-sil-ezra74. fonts-sil-scheherazade

144

D.8 Packages installed on backend 2 and not 3

75. fonts-takao-gothic76. fonts-takao-mincho77. fonts-ukij-uyghur78. fonts-unfonts-core79. g++-4.880. gcc-4.881. gcc-4.8-base:armhf82. gcc-6-base:armhf83. gcc-avr84. gdebi85. gdebi-core86. gecko-mediaplayer87. gfortran-4.888. giblib1:armhf89. gir1.2-clutter-1.090. gir1.2-clutter-gst-2.091. gir1.2-cogl-1.092. gir1.2-coglpango-1.093. gir1.2-freedesktop94. gir1.2-gconf-2.095. gir1.2-gdkpixbuf-2.096. gir1.2-glib-2.097. gir1.2-gtk-3.098. gir1.2-gtkclutter-1.099. gir1.2-gudev-1.0100. gir1.2-ibus-1.0101. gir1.2-javascriptcoregtk-3.0102. gir1.2-json-1.0103. gir1.2-pango-1.0104. gir1.2-polkit-1.0105. gir1.2-rsvg-2.0106. gir1.2-vte-2.90107. gir1.2-webkit-3.0108. gir1.2-wnck-3.0109. gnome-common110. gnome-desktop-data111. gnome-disk-utility112. gnome-icon-theme-full113. gnome-mplayer114. gnome-panel115. gnome-panel-data116. gnome-pkg-tools117. gnome-system-monitor118. gnome-system-tools119. gnumeric

120. gnumeric-common121. gnumeric-doc122. gobject-introspection123. gparted124. gperf125. gpicview126. gsfonts-x11127. gstreamer0.10-nice:armhf128. gstreamer0.10-plugins-base:armhf129. gstreamer0.10-plugins-good:armhf130. gstreamer0.10-x:armhf131. gstreamer1.0-alsa:armhf132. gstreamer1.0-doc133. gstreamer1.0-libav:armhf134. gstreamer1.0-plugins-bad:armhf135. gstreamer1.0-plugins-bad-doc136. gstreamer1.0-plugins-base-apps137. gstreamer1.0-plugins-base-doc138. gstreamer1.0-plugins-good-doc139. gstreamer1.0-tools140. gtk3-engines-unico:armhf141. gtk-doc-tools142. gtk-im-libthai:armhf143. gucharmap144. gvfs-fuse145. hardening-wrapper146. hardinfo147. hunspell-ar148. hunspell-be149. hunspell-da150. hunspell-de-at151. hunspell-de-ch152. hunspell-de-de153. hunspell-eu-es154. hunspell-fr155. hunspell-fr-classical156. hunspell-gl-es157. hunspell-hu158. hunspell-ko159. hunspell-ml160. hunspell-ne161. hunspell-ro162. hunspell-ru163. hunspell-sr164. hunspell-sv-se

145

Chapter D. Backends Specifications

165. hunspell-uz166. hunspell-vi167. hyphen-af168. hyphen-as169. hyphen-bn170. hyphen-ca171. hyphen-de172. hyphen-fr173. hyphen-gu174. hyphen-hi175. hyphen-hr176. hyphen-hu177. hyphen-it178. hyphen-kn179. hyphen-mr180. hyphen-pa181. hyphen-pl182. hyphen-ro183. hyphen-ru184. hyphen-sl185. hyphen-sr186. hyphen-ta187. hyphen-te188. hyphen-zu189. ibus-anthy190. ibus-chewing191. ibus-hangul192. ibus-m17n193. ibus-sunpinyin194. ibus-table195. ibus-table-cangjie3196. ibus-table-cangjie5197. ibus-table-quick-classic198. ibus-table-wubi199. ibus-unikey200. icedtea-7-jre-jamvm:armhf201. indicator-application-gtk2202. intltool203. iotop204. jade205. java-common206. krb5-multidev207. ladspa-sdk208. language-pack-gnome-nb209. language-pack-gnome-nb-base

210. language-pack-nb211. language-pack-nb-base212. leafpad213. liba52-0.7.4214. libaa1-dev215. libabiword-3.0:armhf216. libamd2.3.1:armhf217. libanthy0:armhf218. libapt-inst1.5:armhf219. libapt-pkg4.12:armhf220. libart-2.0-dev221. libasan0:armhf222. libasound2-dev:armhf223. libaspell15224. libasprintf0c2:armhf225. libass4:armhf226. libass-dev:armhf227. libatk1.0-dev228. libatk-bridge2.0-dev:armhf229. libatkmm-1.6-1:armhf230. libatk-wrapper-java231. libatk-wrapper-java-jni:armhf232. libatomic-ops-dev233. libaudclient2:armhf234. libaudcore1:armhf235. libaudit-dev236. libautodie-perl237. libavahi-client-dev238. libavahi-common-dev239. libavahi-glib-dev240. libavc1394-dev:armhf241. libavcodec54:armhf242. libavcodec-dev243. libavformat54:armhf244. libavformat-dev245. libavresample1:armhf246. libavutil52:armhf247. libavutil-dev248. libbinio1ldbl:armhf249. libbison-dev:armhf250. libbluetooth-dev251. libbonobo2-0:armhf252. libbonobo2-common253. libbonobo2-dev:armhf254. libbonoboui2-0:armhf

146

D.8 Packages installed on backend 2 and not 3

255. libbonoboui2-common256. libbonoboui2-dev:armhf257. libboost1.54-dev258. libboost-atomic1.54.0:armhf259. libboost-atomic1.54-dev:armhf260. libboost-chrono1.54.0:armhf261. libboost-chrono1.54-dev:armhf262. libboost-date-time1.54.0:armhf263. libboost-date-time1.54-dev:armhf264. libboost-dev265. libboost-serialization1.54.0:armhf266. libboost-serialization1.54-dev:armhf267. libboost-system1.54.0:armhf268. libboost-system1.54-dev:armhf269. libboost-thread1.54.0:armhf270. libboost-thread1.54-dev:armhf271. libboost-thread-dev:armhf272. libbs2b0273. libbz2-dev:armhf274. libcaca-dev275. libcairo2-dev276. libcairomm-1.0-1:armhf277. libcairo-perl278. libcairo-script-interpreter2:armhf279. libcamd2.3.1:armhf280. libcamel-1.2-45281. libcanberra-dev:armhf282. libcap-dev:armhf283. libcap-ng0284. libccolamd2.8.0:armhf285. libcdaudio1286. libcdaudio-dev287. libcddb2-dev288. libcdio-dev289. libcdparanoia-dev:armhf290. libcdt5291. libcgraph6292. libchamplain-0.12-0:armhf293. libchamplain-gtk-0.12-0:armhf294. libchewing3:armhf295. libchewing3-data:armhf296. libcholmod2.1.2:armhf297. libchromaprint-dev298. libclutter-1.0-dev299. libclutter-gst-2.0-dev

300. libclutter-gtk-1.0-dev301. libcogl15:armhf302. libcogl-dev303. libcogl-pango15:armhf304. libcogl-pango-dev305. libcolord1:armhf306. libcolorhug1:armhf307. libcompfaceg1308. libcue1309. libcurl4-gnutls-dev:armhf310. libcvaux-dev:armhf311. libcv-dev:armhf312. libdaemon0313. libdbus-1-dev:armhf314. libdbus-glib-1-dev315. libdc1394-22-dev:armhf316. libdca-dev:armhf317. libdirac-decoder0:armhf318. libdirac-dev:armhf319. libdirac-encoder0:armhf320. libdirectfb-dev321. libdirectfb-extra:armhf322. libdiscid0:armhf323. libdjvulibre-dev:armhf324. libdmx1:armhf325. libdmx-dev:armhf326. libdv4-dev:armhf327. libdvdnav-dev:armhf328. libdvdread-dev:armhf329. libebook-contacts-1.2-0330. libecal-1.2-16331. libedataserver-1.2-18332. libegl1-mesa-dev333. libegl1-mesa-drivers:armhf334. libelfg0:armhf335. libenca-dev336. libenchant-voikko:armhf337. libept1.4.12:armhf338. libevdocument3-4339. libevview3-3340. libexempi-dev:armhf341. libexif-dev342. libexo-1-0:armhf343. libexo-common344. libexo-helpers

147

Chapter D. Backends Specifications

345. libfaad-dev:armhf346. libfakeroot:armhf347. libfarstream-0.1-0:armhf348. libffi-dev:armhf349. libffmpegthumbnailer4350. libfftw3-bin351. libfftw3-dev:armhf352. libflac-dev:armhf353. libfl-dev:armhf354. libflite1:armhf355. libfluidsynth1:armhf356. libfluidsynth-dev:armhf357. libfm4358. libfm-data359. libfm-extra4360. libfm-gtk4361. libfm-gtk-data362. libfm-modules363. libfontconfig1-dev364. libfontenc-dev:armhf365. libframe6:armhf366. libfreetype6-dev367. libfribidi-dev368. libfs6:armhf369. libftdi1:armhf370. libgail-3-0:armhf371. libgail-common:armhf372. libgail-dev373. libgbm-dev374. libgcc-4.8-dev:armhf375. libgconf2-4:armhf376. libgconf2-dev377. libgcrypt11:armhf378. libgcrypt11-dev379. libgda-5.0-4380. libgda-5.0-common381. libgdk-pixbuf2.0-dev382. libgdome2-0383. libgdome2-cpp-smart0c2a384. libgeis1:armhf385. libgfortran-4.8-dev:armhf386. libgif-dev387. libgirepository-1.0-1388. libgirepository1.0-dev389. libgl1-mesa-dev

390. libglade2-0:armhf391. libgles1-mesa-dev392. libgles2-mesa-dev393. libglib2.0-dev394. libglib2.0-doc395. libglibmm-2.4-1c2a:armhf396. libglib-perl397. libglu1-mesa-dev398. libgme0399. libgme-dev400. libgmlib1:armhf401. libgmp3-dev402. libgmp-dev:armhf403. libgmpxx4ldbl:armhf404. libgmtk1:armhf405. libgmtk1-data406. libgnome2-0:armhf407. libgnome2-bin408. libgnome2-common409. libgnome2-dev:armhf410. libgnome-bluetooth11411. libgnomecanvas2-0:armhf412. libgnomecanvas2-common413. libgnomecanvas2-dev:armhf414. libgnome-desktop-3-7415. libgnome-keyring-dev416. libgnomeui-0:armhf417. libgnomeui-common418. libgnomeui-dev:armhf419. libgnomevfs2-0:armhf420. libgnomevfs2-common421. libgnomevfs2-dev:armhf422. libgnutls26:armhf423. libgnutls-dev424. libgnutlsxx27:armhf425. libgoffice-0.10-10426. libgoffice-0.10-10-common427. libgpg-error-dev428. libgphoto2-port10:armhf429. libgrail6430. libgraphviz-dev431. libgrip0432. libgsf-1-114433. libgsf-1-common434. libgsl0-dev

148

D.8 Packages installed on backend 2 and not 3

435. libgsl0ldbl436. libgsm1-dev:armhf437. libgssrpc4:armhf438. libgstreamer0.10-0:armhf439. libgstreamer1.0-dev440. libgstreamer-plugins-bad1.0-0:armhf441. libgstreamer-plugins-bad1.0-dev442. libgstreamer-plugins-base0.10-

0:armhf443. libgstreamer-plugins-base1.0-dev444. libgstreamer-plugins-good1.0-dev445. libgtk2.0-dev446. libgtk2-perl447. libgtk-3-dev448. libgtkmathview0c2a449. libgtkmm-2.4-1c2a:armhf450. libgtkmm-3.0-1:armhf451. libgtkspell0452. libgtop2-7453. libgucharmap-2-90-7454. libgudev-1.0-dev455. libguess1:armhf456. libgupnp-igd-1.0-4:armhf457. libgvc6458. libgvpr2459. libgweather-3-6460. libhangul1:armhf461. libhangul-data462. libharfbuzz-dev463. libharfbuzz-gobject0:armhf464. libhighgui-dev:armhf465. libhogweed2:armhf466. libical1467. libice-dev:armhf468. libicu52:armhf469. libid3tag0470. libid3tag0-dev471. libidl0:armhf472. libidl-common473. libidl-dev:armhf474. libidn11-dev475. libiec61883-dev476. libijs-0.35477. libilmbase6:armhf478. libilmbase-dev

479. libimage-exiftool-perl480. libimlib2481. libimlib2-dev482. libiptcdata0483. libiptcdata0-dev484. libisl10:armhf485. libisl15:armhf486. libiso9660-dev487. libiw-dev:armhf488. libjack-jackd2-dev:armhf489. libjasper-dev490. libjbig-dev:armhf491. libjna-java492. libjpeg8-dev:armhf493. libjpeg-dev:armhf494. libjpeg-progs495. libjpeg-turbo8-dev:armhf496. libjpeg-turbo-progs497. libjson0:armhf498. libjson-glib-dev499. libkadm5clnt-mit9:armhf500. libkadm5srv-mit9:armhf501. libkate-dev502. libkdb5-7:armhf503. libkrb5-dev504. liblavfile-2.1-0505. liblavjpeg-2.1-0506. liblavplay-2.1-0507. liblcms2-dev:armhf508. libldap2-dev:armhf509. liblink-grammar4510. libllvm3.4:armhf511. liblockfile1:armhf512. liblockfile-bin513. libloudmouth1-0514. liblqr-1-0-dev515. libltdl-dev:armhf516. liblzma-dev:armhf517. liblzo2-dev:armhf518. libm17n-0519. libmad0-dev520. libmagick++5:armhf521. libmagickcore5:armhf522. libmagickcore5-extra:armhf523. libmagickcore-dev

149

Chapter D. Backends Specifications

524. libmagick++-dev525. libmagickwand5:armhf526. libmagickwand-dev527. libmbim-glib0:armhf528. libmeanwhile1529. libmenu-cache3530. libmenu-cache-bin531. libmessaging-menu0532. libmetacity-private0a533. libmicrohttpd-dev534. libmikmod2:armhf535. libmikmod2-dev:armhf536. libmimic0537. libmimic-dev538. libminiupnpc8539. libmirclient7:armhf540. libmirclientplatform-mesa:armhf541. libmirprotobuf0:armhf542. libmirprotobuf-dev:armhf543. libmjpegtools-dev544. libmjpegutils-2.1-0545. libmms0:armhf546. libmms-dev:armhf547. libmodplug1548. libmodplug-dev549. libmowgli2:armhf550. libmp3lame-dev:armhf551. libmpcdec6552. libmpcdec-dev553. libmpeg2-4-dev:armhf554. libmpeg2encpp-2.1-0555. libmpeg3-1556. libmpeg3-dev557. libmpg123-0:armhf558. libmpg123-dev:armhf559. libmplex2-2.1-0560. libmusicbrainz3-6561. libmysqlclient-dev562. libncurses5-dev:armhf563. libnettle4:armhf564. libnfs1:armhf565. libnfs-dev:armhf566. libnice10:armhf567. libnm-glib4568. libnm-glib-vpn1

569. libnm-gtk0570. libnm-util2571. libnotify-bin572. libnotify-dev573. libnss3-1d:armhf574. libntdb1:armhf575. libobrender29576. libobt2577. libofa0578. libofa0-dev579. libogg-dev:armhf580. libonig2581. liboobs-1-5582. libopenal1:armhf583. libopenal-data584. libopenal-dev:armhf585. libopencv2.4-java586. libopencv2.4-jni587. libopencv-calib3d2.4:armhf588. libopencv-calib3d-dev:armhf589. libopencv-contrib2.4:armhf590. libopencv-contrib-dev:armhf591. libopencv-core2.4:armhf592. libopencv-core-dev:armhf593. libopencv-dev594. libopencv-features2d2.4:armhf595. libopencv-features2d-dev:armhf596. libopencv-flann2.4:armhf597. libopencv-flann-dev:armhf598. libopencv-gpu2.4:armhf599. libopencv-gpu-dev:armhf600. libopencv-highgui2.4:armhf601. libopencv-highgui-dev:armhf602. libopencv-imgproc2.4:armhf603. libopencv-imgproc-dev:armhf604. libopencv-legacy2.4:armhf605. libopencv-legacy-dev:armhf606. libopencv-ml2.4:armhf607. libopencv-ml-dev:armhf608. libopencv-objdetect2.4:armhf609. libopencv-objdetect-dev:armhf610. libopencv-ocl2.4:armhf611. libopencv-ocl-dev:armhf612. libopencv-photo2.4:armhf613. libopencv-photo-dev:armhf

150

D.8 Packages installed on backend 2 and not 3

614. libopencv-stitching2.4:armhf615. libopencv-stitching-dev:armhf616. libopencv-superres2.4:armhf617. libopencv-superres-dev:armhf618. libopencv-ts2.4:armhf619. libopencv-ts-dev:armhf620. libopencv-video2.4:armhf621. libopencv-video-dev:armhf622. libopencv-videostab2.4:armhf623. libopencv-videostab-dev:armhf624. libopenexr6:armhf625. libopenexr-dev626. libopenjpeg2:armhf627. libopenjpeg-dev628. libopenvg1-mesa:armhf629. libopts25:armhf630. libopus0631. libopus-dev632. liborbit-2-0:armhf633. liborbit2:armhf634. liborbit2-dev635. liborc-0.4-dev636. libotf0:armhf637. libots0638. libp11-kit-dev639. libpam-cap:armhf640. libpanel-applet-4-0641. libpango1.0-dev642. libpangomm-1.4-1:armhf643. libpango-perl644. libparted0debian1:armhf645. libpathplan4646. libpcre3-dev:armhf647. libpcrecpp0:armhf648. libperl5.18649. libpisock9650. libplist1:armhf651. libplist-dev652. libplymouth2:armhf653. libpng12-dev654. libpolkit-agent-1-dev655. libpolkit-gobject-1-dev656. libpoppler44:armhf657. libpopt-dev:armhf658. libpostproc52

659. libprotobuf8:armhf660. libprotobuf-lite8:armhf661. libproxy1:armhf662. libpulse-dev:armhf663. libpurple0664. libqmi-glib0:armhf665. libqpdf13:armhf666. libquicktime2:armhf667. librarian0668. libraw1394-dev:armhf669. libreadline5:armhf670. libreadline6-dev:armhf671. libreadline-dev:armhf672. librsvg2-dev673. librtmp0:armhf674. librtmp-dev675. librxtx-java676. libsamplerate0-dev:armhf677. libsbc1:armhf678. libsbc-dev:armhf679. libschroedinger-dev:armhf680. libsdl1.2-dev681. libsdl-gfx1.2-4:armhf682. libsdl-gfx1.2-dev:armhf683. libsdl-image1.2-dev:armhf684. libsdl-mixer1.2:armhf685. libsdl-mixer1.2-dev:armhf686. libselinux1-dev:armhf687. libsepol1-dev688. libsgmls-perl689. libshout3-dev:armhf690. libsidplayfp:armhf691. libsigc++-2.0-0c2a:armhf692. libslang2-dev:armhf693. libsmbclient-dev:armhf694. libsm-dev:armhf695. libsndfile1-dev696. libsoundtouch0:armhf697. libsoundtouch-dev698. libsoup2.4-dev699. libsp1c2700. libspandsp2701. libspandsp-dev702. libspeex-dev:armhf703. libsqlite0

151

Chapter D. Backends Specifications

704. libsqlite3-dev:armhf705. libsrtp0706. libsrtp0-dev707. libssh2-1-dev:armhf708. libssh-dev709. libssl-dev:armhf710. libstartup-notification0-dev:armhf711. libstdc++-4.8-dev:armhf712. libsunpinyin3:armhf713. libswscale2:armhf714. libswscale-dev715. libsystemd-daemon0:armhf716. libsystemd-login0:armhf717. libt1-5718. libtag1c2a:armhf719. libtag1-dev720. libtag1-vanilla:armhf721. libtagc0:armhf722. libtagc0-dev723. libtasn1-6-dev724. libtelepathy-glib0:armhf725. libtheora-dev:armhf726. libtidy-0.99-0727. libtiff5-dev:armhf728. libtiffxx5:armhf729. libtinfo-dev:armhf730. libtinyxml2.6.2:armhf731. libtinyxml-dev:armhf732. libts-0.0-0:armhf733. libudev-dev734. libumfpack5.6.2:armhf735. libuniconf4.6736. libupower-glib1:armhf737. libusb-1.0-0-dev:armhf738. libusb-dev739. libusbmuxd2740. libv4l2rds0:armhf741. libv4l-dev:armhf742. libvisual-0.4-dev743. libvncserver0:armhf744. libvo-aacenc0:armhf745. libvo-aacenc-dev:armhf746. libvo-amrwbenc0:armhf747. libvo-amrwbenc-dev:armhf748. libvoikko1:armhf

749. libvorbis-dev:armhf750. libvpx1:armhf751. libvpx-dev:armhf752. libvte-2.90-9753. libvte-2.90-common754. libwavpack-dev:armhf755. libwebpdemux1:armhf756. libwebp-dev:armhf757. libwildmidi1:armhf758. libwildmidi-config759. libwildmidi-dev760. libwmf-dev761. libwpd-0.9-9762. libwpg-0.2-2763. libwps-0.2-2764. libwv-1.2-4:armhf765. libwvstreams4.6-base766. libwvstreams4.6-extras767. libwxbase2.8-0:armhf768. libwxgtk2.8-0:armhf769. libx264-142:armhf770. libxapian22771. libxaw7-dev:armhf772. libxcb-icccm4-dev:armhf773. libxcb-image0-dev:armhf774. libxcb-keysyms1-dev:armhf775. libxcb-shm0-dev:armhf776. libxcb-util0:armhf777. libxcb-util0-dev:armhf778. libxcb-xf86dri0:armhf779. libxcb-xf86dri0-dev:armhf780. libxcb-xv0-dev:armhf781. libxcomposite-dev782. libxcursor-dev:armhf783. libxdot4784. libxfce4ui-1-0785. libxfce4ui-2-0786. libxfce4ui-2-dev787. libxfce4ui-common788. libxfce4util6789. libxfce4util-common790. libxfce4util-dev791. libxfconf-0-2792. libxfconf-0-dev793. libxfont-dev

152

D.8 Packages installed on backend 2 and not 3

794. libxft-dev795. libxi-dev796. libxinerama-dev:armhf797. libxml2-dev:armhf798. libxml2-utils799. libxmu-dev:armhf800. libxmu-headers801. libxmuu-dev:armhf802. libxp6:armhf803. libxpm-dev:armhf804. libxrandr-dev:armhf805. libxrender-dev:armhf806. libxres-dev807. libxslt1-dev:armhf808. libxt-dev:armhf809. libxtst-dev:armhf810. libxv-dev:armhf811. libxvidcore-dev:armhf812. libyajl-dev813. libzbar0814. libzbar-dev815. libzephyr4:armhf816. libzip2817. libzip-dev818. libzvbi-dev:armhf819. light-locker820. light-locker-settings821. link-grammar-dictionaries-en822. localepurge823. lockfile-progs824. lsof825. luatex826. lubuntu-artwork827. lubuntu-artwork-14-04828. lubuntu-icon-theme829. lubuntu-lxpanel-icons830. lubuntu-software-center831. lxappearance832. lxappearance-obconf833. lxde-common834. lxde-core835. lxinput836. lxlauncher837. lxmenu-data838. lxpanel

839. lxpanel-indicator-applet-plugin840. lxrandr841. lxsession842. lxsession-data843. lxsession-default-apps844. lxsession-edit845. lxsession-logout846. lxshortcut847. lxtask848. lxterminal849. lynx850. lynx-cur851. m17n-contrib852. m17n-db853. mc854. mc-data855. medit856. mesa-common-dev857. metacity858. metacity-common859. minicom860. mircommon-dev:armhf861. mountall862. mplayer2863. mtpaint864. nano865. nettle-dev866. ntp867. ntpdate868. obconf869. openbox870. openjdk-7-jre:armhf871. openjdk-7-jre-headless:armhf872. oracle-java8-installer873. orbit2874. pastebinit875. pavucontrol876. pcmanfm877. perl-doc878. pidgin879. pidgin-data880. pidgin-libnotify881. plymouth882. plymouth-label883. plymouth-theme-lubuntu-logo

153

Chapter D. Backends Specifications

884. plymouth-theme-lubuntu-text885. pm-utils886. python-aptdaemon887. python-aptdaemon.gtk3widgets888. python-colorama889. python-commandnotfound890. python-cups891. python-cupshelpers892. python-debian893. python-defer894. python-distlib895. python-gconf896. python-gdbm897. python-glade2898. python-gnomekeyring899. python-gudev900. python-html5lib901. python-libxml2902. python-mako903. python-markupsafe904. python-notify905. python-ntdb906. python-pexpect907. python-pil908. python-pip909. python-psutil910. python-pycurl911. python-pyinotify912. python-pysqlite2913. python-renderpm914. python-reportlab915. python-reportlab-accel916. python-scour917. python-setuptools918. python-smbc919. python-sqlite920. python-support921. python-wheel922. python-xapian923. python-xdg924. rarian-compat925. realpath926. scrot927. sessioninstaller928. sgml-data

929. sgmlspl930. simple-scan931. smbclient932. sp933. sunpinyin-data934. swig935. swig2.0936. sylpheed937. sylpheed-doc938. sylpheed-i18n939. sylpheed-plugins940. synaptic941. systemd-services942. system-tools-backends943. transfig944. transmission945. tsconf946. ttf-bengali-fonts947. ttf-devanagari-fonts948. ttf-gujarati-fonts949. ttf-kannada-fonts950. ttf-malayalam-fonts951. ttf-oriya-fonts952. ttf-punjabi-fonts953. ttf-tamil-fonts954. ttf-telugu-fonts955. tzdata-java956. ubuntu-extras-keyring957. upstart958. valgrind959. voikko-fi960. wbulgarian961. wfrench962. wirish963. witalian964. wmanx965. wnorwegian966. wogerman967. wspanish968. wswedish969. wvdial970. x11proto-bigreqs-dev971. x11proto-composite-dev972. x11proto-dmx-dev973. x11proto-record-dev

154

D.9 Packages installed on backend 3 and not 1

974. x11proto-xcmisc-dev975. x11proto-xf86dga-dev976. x11vnc977. x11vnc-data978. x11-xfs-utils979. xarchiver980. xdg-user-dirs-gtk981. xfburn982. xfce4-dev-tools983. xfce4-notifyd984. xfce4-power-manager985. xfce4-power-manager-data986. xfconf987. xfonts-100dpi988. xmlto

989. xpad990. xscreensaver991. xscreensaver-data992. xscreensaver-data-extra993. xscreensaver-screensaver-bsod994. xserver-xorg-input-multitouch995. xserver-xorg-video-modesetting996. xserver-xorg-video-omap997. xserver-xorg-video-vesa998. xsltproc999. xul-ext-mozvoikko1000. xul-ext-ubufox1001. xvfb1002. yasm

D.9 Packages installed on backend 3 and not 11. adwaita-icon-theme2. apache2-bin3. apg4. aspell5. aspell-en6. atril7. atril-common8. bamfdaemon9. blackbox

10. bluez-obexd11. bootini12. brasero13. brasero-cdrkit14. brasero-common15. caja16. caja-common17. caja-extensions-common18. caja-gksu19. caja-image-converter20. caja-open-terminal21. caja-sendto22. caja-share23. caja-wallpaper24. cgmanager25. cheese-common26. colord-data

27. console-setup-linux28. cpufrequtils29. crda30. cups-pk-helper31. dconf-editor32. deja-dup33. deja-dup-backend-cloudfiles34. deja-dup-backend-gvfs35. deja-dup-backend-s336. deja-dup-caja37. desktop-base38. dmidecode39. dns-root-data40. dosfstools41. duplicity42. dvd+rw-tools43. emacsen-common44. enchant45. energymonitor46. engrampa47. engrampa-common48. eom49. eom-common50. evolution-data-server51. evolution-data-server-online-accounts52. exfat-fuse

155

Chapter D. Backends Specifications

53. exfat-utils54. fbi55. fonts-guru56. fonts-guru-extra57. fonts-lmodern58. fonts-lohit-guru59. fonts-mathjax60. fonts-opensymbol61. fonts-tlwg-laksaman62. gdbserver63. gdisk64. geoclue65. geoclue-ubuntu-geoip66. geoip-database67. gfortran-568. gir1.2-appindicator3-0.169. gir1.2-caja70. gir1.2-freedesktop:armhf71. gir1.2-gdkpixbuf-2.0:armhf72. gir1.2-glib-2.0:armhf73. gir1.2-gtk-3.0:armhf74. gir1.2-ibus-1.0:armhf75. gir1.2-javascriptcoregtk-3.0:armhf76. gir1.2-mate-panel77. gir1.2-pango-1.0:armhf78. gir1.2-peas-1.079. gir1.2-rb-3.080. gir1.2-secret-1:armhf81. gir1.2-vte-2.9182. gir1.2-webkit-3.0:armhf83. gir1.2-wnck-3.0:armhf84. gkbd-capplet85. gnome-bluetooth86. gnome-control-center-shared-data87. gnome-power-manager88. gnome-screensaver89. gnome-session-bin90. gnome-settings-daemon-schemas91. gnome-user-share92. growisofs93. gsettings-ubuntu-schemas94. guile-2.0-libs:armhf95. gvfs-bin96. hddtemp97. hexchat

98. hexchat-common99. hexchat-perl100. hexchat-plugins101. hexchat-python102. humanity-icon-theme103. hwdata104. ideviceinstaller105. ifuse106. imagemagick-6.q16107. indicator-applet108. indicator-bluetooth109. indicator-datetime110. indicator-keyboard111. indicator-messages112. indicator-network113. indicator-power114. indicator-sound115. init116. inxi117. ippusbxd118. iw119. kernel-common120. kodi121. krb5-locales122. liba52-0.7.4:armhf123. libaccount-plugin-1.0-0124. libaccount-plugin-generic-oauth125. libaccount-plugin-google126. libaccounts-glib0:armhf127. libaccounts-qt5-1:armhf128. libalgorithm-c3-perl129. libapache2-mod-dnssd130. libappindicator1131. libapr1:armhf132. libaprutil1:armhf133. libaprutil1-dbd-sqlite3:armhf134. libaprutil1-ldap:armhf135. libapt-inst1.7:armhf136. libapt-pkg4.16:armhf137. libaspell15:armhf138. libasprintf0v5:armhf139. libasprintf-dev:armhf140. libass5:armhf141. libatkmm-1.6-1v5:armhf142. libatm1:armhf

156

D.9 Packages installed on backend 3 and not 1

143. libatrildocument3144. libatrilview3145. libavcodec-ffmpeg56:armhf146. libavformat-ffmpeg56:armhf147. libavutil-ffmpeg54:armhf148. libbabeltrace1:armhf149. libbabeltrace-ctf1:armhf150. libbamf3-2:armhf151. libbareword-filehandles-perl152. libbasicusageenvironment0153. libbdplus0:armhf154. libb-hooks-endofscope-perl155. libb-hooks-op-check-perl156. libblas-common157. libboost-date-time1.58.0:armhf158. libboost-filesystem1.58.0:armhf159. libboost-system1.58.0:armhf160. libbrasero-media3-1161. libbt0v5:armhf162. libcairomm-1.0-1v5:armhf163. libcaja-extension1:armhf164. libcamel-1.2-52165. libcanberra-gtk3-module:armhf166. libcanberra-gtk-module:armhf167. libcanberra-pulse:armhf168. libcap-ng0:armhf169. libcec3:armhf170. libcgi-fast-perl171. libcgi-pm-perl172. libcheese7:armhf173. libcheese-gtk23:armhf174. libclass-c3-perl175. libclass-c3-xs-perl176. libclass-method-modifiers-perl177. libclass-xsaccessor-perl178. libclucene-contribs1v5:armhf179. libclucene-core1v5:armhf180. libclutter-1.0-common181. libcmis-0.5-5v5182. libcogl20:armhf183. libcogl-common184. libcogl-pango20:armhf185. libcogl-path20:armhf186. libcolord2:armhf187. libcolorhug2:armhf

188. libcpan-changes-perl189. libcpan-meta-perl190. libcpufreq0191. libcryptsetup4:armhf192. libdaemon0:armhf193. libdata-optlist-perl194. libdata-perl-perl195. libdata-section-perl196. libdee-1.0-4:armhf197. libdevel-caller-perl198. libdevel-globaldestruction-perl199. libdevel-lexalias-perl200. libdmapsharing-3.0-2201. libdns-export100202. libdouble-conversion1v5:armhf203. libdrm-amdgpu1:armhf204. libdrm-tegra0:armhf205. libdvbpsi10:armhf206. libdw1:armhf207. libebackend-1.2-10208. libebml4v5:armhf209. libebook-1.2-16210. libebook-contacts-1.2-1211. libecal-1.2-18212. libedata-book-1.2-25213. libedata-cal-1.2-27214. libedataserver-1.2-20215. libegl1-mesa-dev:armhf216. libeot0217. libepoxy0218. libevdev2:armhf219. libexiv2-14:armhf220. libexporter-tiny-perl221. libexttextcat-2.0-0222. libexttextcat-data223. libfcgi-perl224. libfcitx-config4:armhf225. libfcitx-gclient0:armhf226. libfcitx-utils0:armhf227. libfdisk1:armhf228. libffmpegthumbnailer4v5229. libfile-desktopentry-perl230. libfile-fcntllock-perl231. libfile-mimeinfo-perl232. libfile-slurp-perl

157

Chapter D. Backends Specifications

233. libfont-afm-perl234. libfreerdp-cache1.1:armhf235. libfreerdp-client1.1:armhf236. libfreerdp-codec1.1:armhf237. libfreerdp-common1.1.0:armhf238. libfreerdp-core1.1:armhf239. libfreerdp-crypto1.1:armhf240. libfreerdp-gdi1.1:armhf241. libfreerdp-locale1.1:armhf242. libfreerdp-primitives1.1:armhf243. libfreerdp-utils1.1:armhf244. libgc1c2:armhf245. libgcrypt20:armhf246. libgdata22:armhf247. libgdata-common248. libgee-0.8-2:armhf249. libgee2:armhf250. libgeocode-glib0:armhf251. libgetopt-long-descriptive-perl252. libgettextpo0:armhf253. libgettextpo-dev:armhf254. libgexiv2-2:armhf255. libgfortran-5-dev:armhf256. libgirepository-1.0-1:armhf257. libgles2-mesa-dev:armhf258. libglew1.10:armhf259. libglibmm-2.4-1v5:armhf260. libgme0:armhf261. libgmime-2.6-0:armhf262. libgnome-bluetooth13263. libgnome-desktop-3-10:armhf264. libgnomekbd8265. libgnomekbd-common266. libgnutls-deb0-28:armhf267. libgoa-1.0-0b:armhf268. libgoa-1.0-common269. libgphoto2-l10n270. libgphoto2-port12:armhf271. libgrilo-0.2-1:armhf272. libgroupsock1273. libgsasl7274. libgsl0ldbl:armhf275. libgtk2.0-bin276. libgtkmm-2.4-1v5:armhf277. libgtksourceview2.0-0

278. libgtksourceview2.0-common279. libgtop2-10280. libguvcview-1.1-1:armhf281. libgweather-3-6:armhf282. libhogweed4:armhf283. libhtml-format-perl284. libhtml-form-perl285. libhttp-daemon-perl286. libhunspell-1.3-0v5:armhf287. libhybris288. libhyphen0289. libical1a290. libicu55:armhf291. libijs-0.35:armhf292. libilmbase12:armhf293. libimobiledevice-utils294. libimport-into-perl295. libindirect-perl296. libinput10:armhf297. libio-stringy-perl298. libirs-export91299. libisccfg-export90300. libisc-export95301. libisl13:armhf302. libjs-mathjax303. libkyotocabinet16v5:armhf304. liblangtag1305. liblangtag-common306. liblcms2-utils307. liblexical-sealrequirehints-perl308. liblivemedia23309. libllvm3.6v5:armhf310. liblog-message-perl311. libmagickcore-6.q16-2:armhf312. libmagickcore-6.q16-2-extra:armhf313. libmagickwand-6.q16-2:armhf314. libmailutils4:armhf315. libmarco-private0:armhf316. libmate-desktop-2-17:armhf317. libmatedict6318. libmatekbd4:armhf319. libmatekbd-common320. libmate-menu2:armhf321. libmatemixer0:armhf322. libmatemixer-common

158

D.9 Packages installed on backend 3 and not 1

323. libmate-panel-applet-4-1324. libmate-sensors-applet-plugin0325. libmate-slab0:armhf326. libmateweather1:armhf327. libmateweather-common328. libmate-window-settings1:armhf329. libmatroska6v5:armhf330. libmbim-glib4:armhf331. libmbim-proxy332. libmedia1333. libminiupnpc10:armhf334. libmirclient9:armhf335. libmircommon5:armhf336. libmircommon-dev:armhf337. libmirprotobuf3:armhf338. libmodplug1:armhf339. libmodule-build-perl340. libmodule-implementation-perl341. libmodule-runtime-perl342. libmodule-signature-perl343. libmoo-perl344. libmoox-handlesvia-perl345. libmpcdec6:armhf346. libmro-compat-perl347. libmultidimensional-perl348. libmythes-1.2-0:armhf349. libnamespace-autoclean-perl350. libnamespace-clean-perl351. libndp0:armhf352. libnettle6:armhf353. libnm0:armhf354. libnm-glib4:armhf355. libnm-glib-vpn1:armhf356. libnm-gtk0:armhf357. libnm-util2:armhf358. libntlm0:armhf359. liboauth0:armhf360. libopenexr22:armhf361. libopenjpeg5:armhf362. libopus0:armhf363. libotr5364. libp8-platform2:armhf365. libpackage-constants-perl366. libpackage-stash-perl367. libpackage-stash-xs-perl

368. libpadwalker-perl369. libpanel-applet0370. libpangomm-1.4-1v5:armhf371. libparams-classify-perl372. libparams-util-perl373. libparams-validate-perl374. libparted2:armhf375. libpath-tiny-perl376. libpcre16-3:armhf377. libpcrecpp0v5:armhf378. libpeas-1.0-0379. libpeas-common380. libperl5.20381. libplank0:armhf382. libplank-common383. libplist3:armhf384. libplist-utils385. libpod-markdown-perl386. libpod-readme-perl387. libpoppler52:armhf388. libpostproc-ffmpeg53:armhf389. libpotrace0390. libprotobuf9v5:armhf391. libprotobuf-lite9v5:armhf392. libproxy1v5:armhf393. libproxy-tools394. libqmi-glib1:armhf395. libqmi-proxy396. libqofono-qt5-0:armhf397. libqpdf13v5:armhf398. libqt5core5a:armhf399. libqt5dbus5:armhf400. libqt5gui5:armhf401. libqt5network5:armhf402. libqt5opengl5:armhf403. libqt5printsupport5:armhf404. libqt5qml5:armhf405. libqt5quick5:armhf406. libqt5sql5:armhf407. libqt5sql5-sqlite:armhf408. libqt5webkit5:armhf409. libqt5widgets5:armhf410. libqt5x11extras5:armhf411. libqt5xml5:armhf412. libqt5xmlpatterns5:armhf

159

Chapter D. Backends Specifications

413. libqwt-headers414. libraw10:armhf415. libreoffice-avmedia-backend-gstreamer416. libreoffice-common417. libreoffice-core418. libreoffice-gnome419. libreoffice-gtk420. libreoffice-l10n-en-za421. libreoffice-style-galaxy422. libreoffice-style-human423. libresid-builder0c2a424. librest-0.7-0:armhf425. librevenge-0.0-0:armhf426. librhythmbox-core9427. librole-tiny-perl428. librsync1:armhf429. librtmp1:armhf430. libruby2.1:armhf431. libsdl2-2.0-0:armhf432. libseccomp2:armhf433. libshine3:armhf434. libsidplay2v5435. libsigc++-2.0-0v5:armhf436. libsignon-extension1:armhf437. libsignon-glib1:armhf438. libsignon-plugins-common1:armhf439. libsignon-qt5-1:armhf440. libsmartcols1:armhf441. libsoftware-license-perl442. libsoxr0:armhf443. libssh-gcrypt-4:armhf444. libstrictures-perl445. libsub-exporter-perl446. libsub-exporter-progressive-perl447. libsub-install-perl448. libswresample-ffmpeg1:armhf449. libswscale-ffmpeg3:armhf450. libsynctex1451. libsystemd0:armhf452. libtag1v5:armhf453. libtag1v5-vanilla:armhf454. libtexlua52455. libtexluajit2456. libtext-template-perl457. libtie-ixhash-perl

458. libtimezonemap1:armhf459. libtimezonemap-data460. libtinyxml2.6.2v5:armhf461. libtotem-plparser18:armhf462. libtotem-plparser-common463. libtry-tiny-perl464. libtwolame0:armhf465. libtxc-dxtn-s2tc0:armhf466. libtype-tiny-perl467. libtype-tiny-xs-perl468. libudev-dev:armhf469. libunicode-utf8-perl470. libunique-1.0-0471. libunity9:armhf472. libunity-control-center1473. libunity-protocol-private0:armhf474. libunity-scopes-json-def-desktop475. libunity-settings-daemon1476. libunwind8477. libupnp6478. libupower-glib3:armhf479. liburl-dispatcher1:armhf480. libusageenvironment1481. libusbmuxd2:armhf482. libva-drm1:armhf483. libvariable-magic-perl484. libva-x11-1:armhf485. libvcdinfo0486. libvisual-0.4-plugins:armhf487. libvlc5488. libvlccore8489. libvncclient1:armhf490. libvpx2:armhf491. libvte-2.91-0492. libvte-2.91-common493. libwacom2:armhf494. libwacom-bin495. libwacom-common496. libwebrtc-audio-processing-0:armhf497. libwinpr-crt0.1:armhf498. libwinpr-dsparse0.1:armhf499. libwinpr-environment0.1:armhf500. libwinpr-file0.1:armhf501. libwinpr-handle0.1:armhf502. libwinpr-heap0.1:armhf

160

D.9 Packages installed on backend 3 and not 1

503. libwinpr-input0.1:armhf504. libwinpr-interlocked0.1:armhf505. libwinpr-library0.1:armhf506. libwinpr-path0.1:armhf507. libwinpr-pool0.1:armhf508. libwinpr-registry0.1:armhf509. libwinpr-rpc0.1:armhf510. libwinpr-sspi0.1:armhf511. libwinpr-synch0.1:armhf512. libwinpr-sysinfo0.1:armhf513. libwinpr-thread0.1:armhf514. libwinpr-utils0.1:armhf515. libx11-protocol-perl516. libx264-146:armhf517. libx265-59:armhf518. libxapian22v5519. libxcb-composite0:armhf520. libxcb-render-util0:armhf521. libxcb-util1:armhf522. libxcb-xkb1:armhf523. libxkbcommon-x11-0:armhf524. libxml-xpathengine-perl525. libyaml-0-2:armhf526. libzeitgeist-2.0-0:armhf527. libzip4:armhf528. libzzip-0-13:armhf529. lightdm-gtk-greeter-settings530. linux-image-3.10.96-78531. linux-image-xu3532. linux-tools-4.2.0-23533. linux-tools-4.2.0-23-generic-lpae534. linux-tools-common535. linux-tools-generic-lpae536. live-boot-initramfs-tools537. lmodern538. mailutils539. mailutils-common540. mali-x11541. marco542. marco-common543. mate-applets544. mate-applets-common545. mate-backgrounds546. mate-control-center547. mate-control-center-common

548. mate-desktop549. mate-desktop-common550. mate-desktop-environment551. mate-desktop-environment-core552. mate-desktop-environment-extra553. mate-desktop-environment-extras554. mate-gnome-main-menu-applet555. mate-icon-theme556. mate-icon-theme-faenza557. mate-indicator-applet558. mate-indicator-applet-common559. mate-media560. mate-media-common561. mate-menus562. mate-netspeed563. mate-netspeed-common564. mate-notification-daemon565. mate-notification-daemon-common566. mate-panel567. mate-panel-common568. mate-polkit:armhf569. mate-polkit-common570. mate-power-manager571. mate-power-manager-common572. mate-screensaver573. mate-screensaver-common574. mate-sensors-applet575. mate-sensors-applet-common576. mate-session-manager577. mate-settings-daemon578. mate-settings-daemon-common579. mate-system-monitor580. mate-system-monitor-common581. mate-terminal582. mate-terminal-common583. mate-themes584. mate-user-guide585. mate-user-share586. mate-user-share-common587. mate-utils588. mate-utils-common589. media-player-info590. menu591. menu-xdg592. mesa-common-dev:armhf

161

Chapter D. Backends Specifications

593. mir-client-platform-mesa-dev:armhf594. mousetweaks595. mozo596. myspell-en-au597. myspell-en-gb598. myspell-en-za599. mythes-en-au600. mythes-en-us601. ncurses-term602. network-manager-pptp603. network-manager-pptp-gnome604. ntfs-3g605. odroid-platform-5422606. ofono607. openoffice.org-hyphenation608. p7zip-full609. pidgin-otr610. pinentry-gnome3611. plank612. pluma613. pluma-common614. postfix615. pptp-linux616. printer-driver-brlaser617. python3-bs4618. python3-cairo619. python3-cups620. python3-cupshelpers621. python3-html5lib622. python3-lxml623. python3-mako624. python3-markupsafe625. python3-pexpect626. python3-pil:armhf627. python3-pyinotify628. python3-renderpm:armhf629. python3-reportlab630. python3-reportlab-accel:armhf631. python3-requests632. python3-smbc633. python3-systemd634. python3-uno635. python3-urllib3636. python3-virtualenv637. python3-xdg

638. python-boto639. python-caja640. python-caja-common641. python-cffi642. python-cffi-backend643. python-cloudfiles644. python-cryptography645. python-enum34646. python-gtksourceview2647. python-idna648. python-ipaddress649. python-lockfile650. python-mate-menu651. python-ndg-httpsclient652. python-openssl653. python-pil:armhf654. python-ply655. python-pyasn1656. python-pycparser657. qttranslations5-l10n658. rename659. rhythmbox660. rhythmbox-data661. rhythmbox-plugin-cdrecorder662. rhythmbox-plugins663. rhythmbox-plugin-zeitgeist664. rtkit665. ruby666. ruby2.1667. rubygems-integration668. seahorse669. session-migration670. shotwell671. shotwell-common672. signond673. signon-keyring-extension674. signon-plugin-oauth2675. signon-ui676. signon-ui-service677. signon-ui-x11678. smartmontools679. ssh-import-id680. systemd681. systemd-sysv682. tcpd

162

D.10 Packages installed on backend 3 and not 2

683. tcptrack684. telnet685. thunderbird686. thunderbird-locale-en687. thunderbird-locale-en-gb688. thunderbird-locale-en-us689. tlp690. tlp-rdw691. traceroute692. ttf-ancient-fonts-symbola693. ubuntu-mate-icon-themes694. ubuntu-mate-lightdm-theme695. ubuntu-mate-themes696. ubuntu-mate-wallpapers697. ubuntu-mate-wallpapers-common698. ubuntu-mate-wallpapers-wily699. ubuntu-minimal700. ubuntu-mobile-icons701. ubuntu-mono702. ubuntu-system-service703. ubuntu-touch-sounds704. unity-control-center705. unity-control-center-signon

706. unity-settings-daemon707. uno-libs3708. ure709. ureadahead710. urfkill711. va-driver-all:armhf712. vdpau-va-driver:armhf713. virtualenv714. vlc715. vlc-data716. vlc-nox717. vlc-plugin-notify718. vlc-plugin-samba719. wireless-regdb720. wodim721. xserver-xorg-core722. xserver-xorg-input-wacom723. xserver-xorg-video-armsoc-5422724. xul-ext-calendar-timezones725. xul-ext-gdata-provider726. xul-ext-lightning727. zeitgeist-core

D.10 Packages installed on backend 3 and not 21. adwaita-icon-theme2. apache2-bin3. apg4. atril5. atril-common6. bamfdaemon7. blackbox8. bluez-obexd9. bootini

10. brasero11. brasero-cdrkit12. brasero-common13. caja14. caja-common15. caja-extensions-common16. caja-gksu17. caja-image-converter18. caja-open-terminal

19. caja-sendto20. caja-share21. caja-wallpaper22. cgmanager23. cheese-common24. colord-data25. console-setup-linux26. cpufrequtils27. crda28. cups-pk-helper29. dconf-editor30. deja-dup31. deja-dup-backend-cloudfiles32. deja-dup-backend-gvfs33. deja-dup-backend-s334. deja-dup-caja35. desktop-base36. dmidecode

163

Chapter D. Backends Specifications

37. dns-root-data38. dosfstools39. duplicity40. dvd+rw-tools41. emacsen-common42. enchant43. energymonitor44. engrampa45. engrampa-common46. eom47. eom-common48. evolution-data-server49. evolution-data-server-online-accounts50. exfat-fuse51. exfat-utils52. fbi53. fonts-guru54. fonts-guru-extra55. fonts-lmodern56. fonts-lohit-guru57. fonts-mathjax58. fonts-opensymbol59. fonts-tlwg-laksaman60. gdbserver61. gdisk62. geoclue63. geoclue-ubuntu-geoip64. geoip-database65. gfortran-566. gir1.2-appindicator3-0.167. gir1.2-caja68. gir1.2-freedesktop:armhf69. gir1.2-gdkpixbuf-2.0:armhf70. gir1.2-glib-2.0:armhf71. gir1.2-gtk-3.0:armhf72. gir1.2-ibus-1.0:armhf73. gir1.2-javascriptcoregtk-3.0:armhf74. gir1.2-mate-panel75. gir1.2-pango-1.0:armhf76. gir1.2-peas-1.077. gir1.2-rb-3.078. gir1.2-secret-1:armhf79. gir1.2-vte-2.9180. gir1.2-webkit-3.0:armhf81. gir1.2-wnck-3.0:armhf

82. gkbd-capplet83. gnome-bluetooth84. gnome-control-center-shared-data85. gnome-power-manager86. gnome-screensaver87. gnome-session-bin88. gnome-settings-daemon-schemas89. gnome-user-share90. growisofs91. gsettings-ubuntu-schemas92. guile-2.0-libs:armhf93. gvfs-bin94. hddtemp95. hexchat96. hexchat-common97. hexchat-perl98. hexchat-plugins99. hexchat-python100. humanity-icon-theme101. hwdata102. ideviceinstaller103. ifuse104. imagemagick105. imagemagick-6.q16106. indicator-applet107. indicator-bluetooth108. indicator-datetime109. indicator-keyboard110. indicator-messages111. indicator-network112. indicator-power113. indicator-sound114. init115. inxi116. ippusbxd117. iw118. kernel-common119. kodi120. krb5-locales121. liba52-0.7.4:armhf122. libaccount-plugin-1.0-0123. libaccount-plugin-generic-oauth124. libaccount-plugin-google125. libaccounts-glib0:armhf126. libaccounts-qt5-1:armhf

164

D.10 Packages installed on backend 3 and not 2

127. libalgorithm-c3-perl128. libapache2-mod-dnssd129. libappindicator1130. libapr1:armhf131. libaprutil1:armhf132. libaprutil1-dbd-sqlite3:armhf133. libaprutil1-ldap:armhf134. libapt-inst1.7:armhf135. libapt-pkg4.16:armhf136. libaspell15:armhf137. libasprintf0v5:armhf138. libasprintf-dev:armhf139. libass5:armhf140. libatkmm-1.6-1v5:armhf141. libatm1:armhf142. libatrildocument3143. libatrilview3144. libavcodec-ffmpeg56:armhf145. libavformat-ffmpeg56:armhf146. libavutil-ffmpeg54:armhf147. libbabeltrace1:armhf148. libbabeltrace-ctf1:armhf149. libbamf3-2:armhf150. libbareword-filehandles-perl151. libbasicusageenvironment0152. libbdplus0:armhf153. libb-hooks-endofscope-perl154. libb-hooks-op-check-perl155. libblas-common156. libboost-date-time1.58.0:armhf157. libboost-filesystem1.58.0:armhf158. libboost-system1.58.0:armhf159. libbrasero-media3-1160. libbt0v5:armhf161. libcairomm-1.0-1v5:armhf162. libcaja-extension1:armhf163. libcamel-1.2-52164. libcanberra-gtk3-module:armhf165. libcanberra-gtk-module:armhf166. libcanberra-pulse:armhf167. libcap-ng0:armhf168. libcec3:armhf169. libcgi-fast-perl170. libcgi-pm-perl171. libcheese7:armhf

172. libcheese-gtk23:armhf173. libclass-c3-perl174. libclass-c3-xs-perl175. libclass-method-modifiers-perl176. libclass-xsaccessor-perl177. libclucene-contribs1v5:armhf178. libclucene-core1v5:armhf179. libclutter-1.0-common180. libcmis-0.5-5v5181. libcogl20:armhf182. libcogl-common183. libcogl-pango20:armhf184. libcogl-path20:armhf185. libcolord2:armhf186. libcolorhug2:armhf187. libcpan-changes-perl188. libcpan-meta-perl189. libcpufreq0190. libcryptsetup4:armhf191. libdaemon0:armhf192. libdata-optlist-perl193. libdata-perl-perl194. libdata-section-perl195. libdee-1.0-4:armhf196. libdevel-caller-perl197. libdevel-globaldestruction-perl198. libdevel-lexalias-perl199. libdmapsharing-3.0-2200. libdns-export100201. libdouble-conversion1v5:armhf202. libdrm-amdgpu1:armhf203. libdrm-tegra0:armhf204. libdvbpsi10:armhf205. libdw1:armhf206. libebackend-1.2-10207. libebml4v5:armhf208. libebook-1.2-16209. libebook-contacts-1.2-1210. libecal-1.2-18211. libedata-book-1.2-25212. libedata-cal-1.2-27213. libedataserver-1.2-20214. libegl1-mesa-dev:armhf215. libeot0216. libepoxy0

165

Chapter D. Backends Specifications

217. libevdev2:armhf218. libexiv2-14:armhf219. libexporter-tiny-perl220. libexttextcat-2.0-0221. libexttextcat-data222. libfcgi-perl223. libfcitx-config4:armhf224. libfcitx-gclient0:armhf225. libfcitx-utils0:armhf226. libfdisk1:armhf227. libffmpegthumbnailer4v5228. libfile-desktopentry-perl229. libfile-fcntllock-perl230. libfile-mimeinfo-perl231. libfile-slurp-perl232. libfont-afm-perl233. libfreerdp-cache1.1:armhf234. libfreerdp-client1.1:armhf235. libfreerdp-codec1.1:armhf236. libfreerdp-common1.1.0:armhf237. libfreerdp-core1.1:armhf238. libfreerdp-crypto1.1:armhf239. libfreerdp-gdi1.1:armhf240. libfreerdp-locale1.1:armhf241. libfreerdp-primitives1.1:armhf242. libfreerdp-utils1.1:armhf243. libgc1c2:armhf244. libgcrypt20:armhf245. libgdata22:armhf246. libgdata-common247. libgee-0.8-2:armhf248. libgee2:armhf249. libgeocode-glib0:armhf250. libgetopt-long-descriptive-perl251. libgettextpo0:armhf252. libgettextpo-dev:armhf253. libgexiv2-2:armhf254. libgfortran-5-dev:armhf255. libgirepository-1.0-1:armhf256. libgles2-mesa-dev:armhf257. libglew1.10:armhf258. libglibmm-2.4-1v5:armhf259. libgme0:armhf260. libgmime-2.6-0:armhf261. libgnome-bluetooth13

262. libgnome-desktop-3-10:armhf263. libgnomekbd8264. libgnomekbd-common265. libgnutls-deb0-28:armhf266. libgoa-1.0-0b:armhf267. libgoa-1.0-common268. libgphoto2-l10n269. libgphoto2-port12:armhf270. libgrilo-0.2-1:armhf271. libgroupsock1272. libgsasl7273. libgsl0ldbl:armhf274. libgtk2.0-bin275. libgtkmm-2.4-1v5:armhf276. libgtksourceview2.0-0277. libgtksourceview2.0-common278. libgtop2-10279. libguvcview-1.1-1:armhf280. libgweather-3-6:armhf281. libhogweed4:armhf282. libhtml-format-perl283. libhtml-form-perl284. libhttp-daemon-perl285. libhunspell-1.3-0v5:armhf286. libhybris287. libhyphen0288. libical1a289. libicu55:armhf290. libijs-0.35:armhf291. libilmbase12:armhf292. libimobiledevice-utils293. libimport-into-perl294. libindirect-perl295. libinput10:armhf296. libio-stringy-perl297. libirs-export91298. libisccfg-export90299. libisc-export95300. libisl13:armhf301. libjavascriptcoregtk-1.0-0:armhf302. libjs-mathjax303. libkyotocabinet16v5:armhf304. liblangtag1305. liblangtag-common306. liblcms2-utils

166

D.10 Packages installed on backend 3 and not 2

307. liblexical-sealrequirehints-perl308. liblivemedia23309. libllvm3.6v5:armhf310. liblog-message-perl311. liblua5.1-0:armhf312. libmagickcore-6.q16-2:armhf313. libmagickcore-6.q16-2-extra:armhf314. libmagickwand-6.q16-2:armhf315. libmailutils4:armhf316. libmarco-private0:armhf317. libmate-desktop-2-17:armhf318. libmatedict6319. libmatekbd4:armhf320. libmatekbd-common321. libmate-menu2:armhf322. libmatemixer0:armhf323. libmatemixer-common324. libmate-panel-applet-4-1325. libmate-sensors-applet-plugin0326. libmate-slab0:armhf327. libmateweather1:armhf328. libmateweather-common329. libmate-window-settings1:armhf330. libmatroska6v5:armhf331. libmbim-glib4:armhf332. libmbim-proxy333. libmedia1334. libminiupnpc10:armhf335. libmirclient9:armhf336. libmircommon5:armhf337. libmircommon-dev:armhf338. libmirprotobuf3:armhf339. libmng2:armhf340. libmodplug1:armhf341. libmodule-build-perl342. libmodule-implementation-perl343. libmodule-runtime-perl344. libmodule-signature-perl345. libmoo-perl346. libmoox-handlesvia-perl347. libmpcdec6:armhf348. libmro-compat-perl349. libmultidimensional-perl350. libmythes-1.2-0:armhf351. libnamespace-autoclean-perl

352. libnamespace-clean-perl353. libndp0:armhf354. libnettle6:armhf355. libnm0:armhf356. libnm-glib4:armhf357. libnm-glib-vpn1:armhf358. libnm-gtk0:armhf359. libnm-util2:armhf360. libntlm0:armhf361. liboauth0:armhf362. libopenexr22:armhf363. libopenjpeg5:armhf364. libopus0:armhf365. libotr5366. libp8-platform2:armhf367. libpackage-constants-perl368. libpackage-stash-perl369. libpackage-stash-xs-perl370. libpadwalker-perl371. libpanel-applet0372. libpangomm-1.4-1v5:armhf373. libparams-classify-perl374. libparams-util-perl375. libparams-validate-perl376. libparted2:armhf377. libpath-tiny-perl378. libpcre16-3:armhf379. libpcrecpp0v5:armhf380. libpeas-1.0-0381. libpeas-common382. libperl5.20383. libplank0:armhf384. libplank-common385. libplist3:armhf386. libplist-utils387. libpod-markdown-perl388. libpod-readme-perl389. libpoppler52:armhf390. libpostproc-ffmpeg53:armhf391. libpotrace0392. libprotobuf9v5:armhf393. libprotobuf-lite9v5:armhf394. libproxy1v5:armhf395. libproxy-tools396. libqmi-glib1:armhf

167

Chapter D. Backends Specifications

397. libqmi-proxy398. libqofono-qt5-0:armhf399. libqpdf13v5:armhf400. libqt5core5a:armhf401. libqt5dbus5:armhf402. libqt5gui5:armhf403. libqt5network5:armhf404. libqt5opengl5:armhf405. libqt5printsupport5:armhf406. libqt5qml5:armhf407. libqt5quick5:armhf408. libqt5sql5:armhf409. libqt5sql5-sqlite:armhf410. libqt5webkit5:armhf411. libqt5widgets5:armhf412. libqt5x11extras5:armhf413. libqt5xml5:armhf414. libqt5xmlpatterns5:armhf415. libqwt-headers416. libraw10:armhf417. libreoffice-avmedia-backend-gstreamer418. libreoffice-common419. libreoffice-core420. libreoffice-gnome421. libreoffice-gtk422. libreoffice-l10n-en-za423. libreoffice-style-galaxy424. libreoffice-style-human425. libresid-builder0c2a426. librest-0.7-0:armhf427. librevenge-0.0-0:armhf428. librhythmbox-core9429. librole-tiny-perl430. librsync1:armhf431. librtmp1:armhf432. libruby2.1:armhf433. libsdl2-2.0-0:armhf434. libseccomp2:armhf435. libshine3:armhf436. libsidplay2v5437. libsigc++-2.0-0v5:armhf438. libsignon-extension1:armhf439. libsignon-glib1:armhf440. libsignon-plugins-common1:armhf441. libsignon-qt5-1:armhf

442. libsmartcols1:armhf443. libsoftware-license-perl444. libsoxr0:armhf445. libssh-gcrypt-4:armhf446. libstrictures-perl447. libsub-exporter-perl448. libsub-exporter-progressive-perl449. libsub-install-perl450. libswresample-ffmpeg1:armhf451. libswscale-ffmpeg3:armhf452. libsynctex1453. libsystemd0:armhf454. libtag1v5:armhf455. libtag1v5-vanilla:armhf456. libtexlua52457. libtexluajit2458. libtext-template-perl459. libtie-ixhash-perl460. libtimezonemap1:armhf461. libtimezonemap-data462. libtinyxml2.6.2v5:armhf463. libtotem-plparser18:armhf464. libtotem-plparser-common465. libtry-tiny-perl466. libtwolame0:armhf467. libtxc-dxtn-s2tc0:armhf468. libtype-tiny-perl469. libtype-tiny-xs-perl470. libudev-dev:armhf471. libunicode-utf8-perl472. libunique-1.0-0473. libunity9:armhf474. libunity-control-center1475. libunity-protocol-private0:armhf476. libunity-scopes-json-def-desktop477. libunity-settings-daemon1478. libunwind8479. libupnp6480. libupower-glib3:armhf481. liburl-dispatcher1:armhf482. libusageenvironment1483. libusbmuxd2:armhf484. libva-drm1:armhf485. libvariable-magic-perl486. libva-x11-1:armhf

168

D.10 Packages installed on backend 3 and not 2

487. libvcdinfo0488. libvisual-0.4-plugins:armhf489. libvlc5490. libvlccore8491. libvncclient1:armhf492. libvpx2:armhf493. libvte-2.91-0494. libvte-2.91-common495. libwacom2:armhf496. libwacom-bin497. libwacom-common498. libwebkitgtk-1.0-0:armhf499. libwebkitgtk-1.0-common500. libwebrtc-audio-processing-0:armhf501. libwinpr-crt0.1:armhf502. libwinpr-dsparse0.1:armhf503. libwinpr-environment0.1:armhf504. libwinpr-file0.1:armhf505. libwinpr-handle0.1:armhf506. libwinpr-heap0.1:armhf507. libwinpr-input0.1:armhf508. libwinpr-interlocked0.1:armhf509. libwinpr-library0.1:armhf510. libwinpr-path0.1:armhf511. libwinpr-pool0.1:armhf512. libwinpr-registry0.1:armhf513. libwinpr-rpc0.1:armhf514. libwinpr-sspi0.1:armhf515. libwinpr-synch0.1:armhf516. libwinpr-sysinfo0.1:armhf517. libwinpr-thread0.1:armhf518. libwinpr-utils0.1:armhf519. libx11-protocol-perl520. libx264-146:armhf521. libx265-59:armhf522. libxapian22v5523. libxcb-composite0:armhf524. libxcb-render-util0:armhf525. libxcb-util1:armhf526. libxcb-xkb1:armhf527. libxkbcommon-x11-0:armhf528. libxml-xpathengine-perl529. libyaml-0-2:armhf530. libzeitgeist-2.0-0:armhf531. libzip4:armhf

532. libzzip-0-13:armhf533. lightdm-gtk-greeter-settings534. linux-image-3.10.96-78535. linux-image-xu3536. linux-tools-4.2.0-23537. linux-tools-4.2.0-23-generic-lpae538. linux-tools-common539. linux-tools-generic-lpae540. live-boot-initramfs-tools541. lmodern542. mailutils543. mailutils-common544. mali-x11545. marco546. marco-common547. mate-applets548. mate-applets-common549. mate-backgrounds550. mate-control-center551. mate-control-center-common552. mate-desktop553. mate-desktop-common554. mate-desktop-environment555. mate-desktop-environment-core556. mate-desktop-environment-extra557. mate-desktop-environment-extras558. mate-gnome-main-menu-applet559. mate-icon-theme560. mate-icon-theme-faenza561. mate-indicator-applet562. mate-indicator-applet-common563. mate-media564. mate-media-common565. mate-menus566. mate-netspeed567. mate-netspeed-common568. mate-notification-daemon569. mate-notification-daemon-common570. mate-panel571. mate-panel-common572. mate-polkit:armhf573. mate-polkit-common574. mate-power-manager575. mate-power-manager-common576. mate-screensaver

169

Chapter D. Backends Specifications

577. mate-screensaver-common578. mate-sensors-applet579. mate-sensors-applet-common580. mate-session-manager581. mate-settings-daemon582. mate-settings-daemon-common583. mate-system-monitor584. mate-system-monitor-common585. mate-terminal586. mate-terminal-common587. mate-themes588. mate-user-guide589. mate-user-share590. mate-user-share-common591. mate-utils592. mate-utils-common593. media-player-info594. menu595. menu-xdg596. mesa-common-dev:armhf597. mir-client-platform-mesa-dev:armhf598. mousetweaks599. mozo600. myspell-en-au601. myspell-en-gb602. myspell-en-za603. mythes-en-au604. mythes-en-us605. ncurses-term606. network-manager-pptp607. network-manager-pptp-gnome608. ntfs-3g609. odroid-platform-5422610. ofono611. openoffice.org-hyphenation612. p7zip-full613. pidgin-otr614. pinentry-gnome3615. plank616. pluma617. pluma-common618. postfix619. pptp-linux620. printer-driver-brlaser621. python3-bs4

622. python3-cairo623. python3-cups624. python3-cupshelpers625. python3-html5lib626. python3-lxml627. python3-mako628. python3-markupsafe629. python3-pexpect630. python3-pil:armhf631. python3-pyinotify632. python3-renderpm:armhf633. python3-reportlab634. python3-reportlab-accel:armhf635. python3-requests636. python3-smbc637. python3-systemd638. python3-uno639. python3-urllib3640. python3-virtualenv641. python3-xdg642. python-boto643. python-caja644. python-caja-common645. python-cffi646. python-cffi-backend647. python-cloudfiles648. python-cryptography649. python-enum34650. python-gtksourceview2651. python-idna652. python-ipaddress653. python-lockfile654. python-mate-menu655. python-ndg-httpsclient656. python-openssl657. python-pil:armhf658. python-ply659. python-pyasn1660. python-pycparser661. qttranslations5-l10n662. rename663. rhythmbox664. rhythmbox-data665. rhythmbox-plugin-cdrecorder666. rhythmbox-plugins

170

D.10 Packages installed on backend 3 and not 2

667. rhythmbox-plugin-zeitgeist668. rtkit669. ruby670. ruby2.1671. rubygems-integration672. seahorse673. session-migration674. shotwell675. shotwell-common676. signond677. signon-keyring-extension678. signon-plugin-oauth2679. signon-ui680. signon-ui-service681. signon-ui-x11682. smartmontools683. ssh-import-id684. systemd685. systemd-sysv686. tcpd687. tcptrack688. telnet689. texlive-base690. thunderbird691. thunderbird-locale-en692. thunderbird-locale-en-gb693. thunderbird-locale-en-us694. tlp695. tlp-rdw696. traceroute697. ttf-ancient-fonts-symbola698. ubuntu-mate-icon-themes699. ubuntu-mate-lightdm-theme

700. ubuntu-mate-themes701. ubuntu-mate-wallpapers702. ubuntu-mate-wallpapers-common703. ubuntu-mate-wallpapers-wily704. ubuntu-minimal705. ubuntu-mobile-icons706. ubuntu-mono707. ubuntu-system-service708. ubuntu-touch-sounds709. unity-control-center710. unity-control-center-signon711. unity-settings-daemon712. uno-libs3713. ure714. ureadahead715. urfkill716. va-driver-all:armhf717. vdpau-va-driver:armhf718. virtualenv719. vlc720. vlc-data721. vlc-nox722. vlc-plugin-notify723. vlc-plugin-samba724. wireless-regdb725. wodim726. xserver-xorg-core727. xserver-xorg-input-wacom728. xserver-xorg-video-armsoc-5422729. xul-ext-calendar-timezones730. xul-ext-gdata-provider731. xul-ext-lightning732. zeitgeist-core

171