autonomic workload management for database ... - QSpace

AUTONOMIC WORKLOAD MANAGEMENT FOR

DATABASE MANAGEMENT SYSTEMS

by

MINGYI ZHANG

A thesis submitted to the School of Computing

In conformity with the requirements for

the degree of Doctor of Philosophy

Queen‟s University

Kingston, Ontario, Canada

April, 2014

Copyright © Mingyi Zhang, 2014

ii

Abstract

In today‟s database server environments, multiple types of workloads, such as on-line

transaction processing, business intelligence and administrative utilities, can be present in a

system simultaneously. Workloads may have different levels of business importance and distinct

performance objectives. When the workloads execute concurrently on a database server,

interference may occur and result in the workloads failing to meet the performance objectives and

the database server suffering severe performance degradation.

To evaluate and classify the existing workload management systems and techniques, we

develop a taxonomy of workload management techniques. The taxonomy categorizes workload

management techniques into multiple classes and illustrates a workload management process.

We propose a general framework for autonomic workload management for database

management systems (DBMSs) to dynamically monitor and control the flow of the workloads

and help DBMSs achieve the performance objectives without human intervention. Our

framework consists of multiple workload management techniques and performance monitor

functions, and implements the monitor–analyze–plan–execute loop suggested in autonomic

computing principles. When a performance issue arises, our framework provides the ability to

dynamically detect the issue and to initiate and coordinate the workload management techniques.

To detect severe performance degradation in database systems, we propose the use of

indicators. We demonstrate a learning-based approach to identify a set of internal DBMS monitor

metrics that best indicate the problem. We illustrate and validate our framework and approaches

using a prototype system implemented on top of IBM DB2 Workload Manager. Our prototype

system leverages the existing workload management facilities and implements a set of

corresponding controllers to adapt to dynamic and mixed workloads while protecting DBMSs

against severe performance degradation.

iii

Acknowledgements

I would like to extend my sincerest thanks to my supervisor, Dr. Patrick Martin, for

providing me the opportunity to pursue my research interest in the areas of database management

systems and autonomic computing. I am heartily thankful to Dr. Martin for all his great guidance,

help and support throughout these years of my graduate studies (PhDs and Masters) in the School

of Computing at Queen's University in Kingston, Canada.

I would like to heartily thank Wendy Powley for her great help during these years when I

was pursuing my research at Database Systems Laboratory in the school. Without Dr. Martin‟s

and Wendy‟s guidance and help, this research would be impossible.

I would like to thank Dr. Juergen Dingel and Dr. Ahmed Hassan for serving on my

supervisory committee, and for their feedback on my progress reports, breadth proposal, depth

proposal, depth paper and thesis proposal.

It was a great pleasure to work with the talented and bright people at IBM. I would like to

thank Paul Bird, Keith McDonald and David Kalmuk at IBM Toronto Lab for their comments

and advice, and Joanna Ng, Calisto Zuzarte, Emilia Tung, Jimmy Lo and all people at Center for

Advanced Studies at IBM Toronto Lab for their help with this research. Additional thanks to the

members of Database Systems Laboratory for their opinions and friendship.

I would like to thank Center for Advanced Studies at IBM Toronto Lab, Canada, Natural

Science and Engineering Research Council of Canada, and Queen's University in Kingston,

Canada for their financial support.

Finally, I would like to thank my parents, my sister and brother-in-law, my wife and my

nephew for their love, encouragement and support. I would also like to extend a deep gratitude to

all my friends.

iv

Statement of Originality

I hereby certify that all of the work described within this thesis is the original work of the author.

Any published (or unpublished) ideas and/or techniques from the work of others are fully

acknowledged in accordance with the standard referencing practices.

(Mingyi Zhang)

(January, 2014)

v

Table of Contents

Abstract ............................................................................................................................................ ii

Acknowledgements ......................................................................................................................... iii

Statement of Originality .................................................................................................................. iv

Table of Contents ............................................................................................................................. v

List of Figures ................................................................................................................................ vii

List of Tables .................................................................................................................................. ix

List of Acronyms ............................................................................................................................. x

Chapter 1 Introduction ..................................................................................................................... 1

1.1 Motivation .............................................................................................................................. 2

1.2 Thesis Statement .................................................................................................................... 5

1.3 Contributions ......................................................................................................................... 6

1.4 Outline ................................................................................................................................... 8

Chapter 2 Literature Review ............................................................................................................ 9

2.1 Autonomic Computing ........................................................................................................... 9

2.2 Workload Management ........................................................................................................ 12

2.2.1 Performance Objectives and Management Policies ...................................................... 13

2.2.2 Request Identification ................................................................................................... 15

2.2.3 Workload Control ......................................................................................................... 16

2.3 A Taxonomy of Workload Management Techniques .......................................................... 19

2.3.1 Workload Characterization ........................................................................................... 20

2.3.2 Admission Control ........................................................................................................ 22

2.3.3 Scheduling..................................................................................................................... 24

2.3.4 Execution Control ......................................................................................................... 26

2.4 Autonomic Workload Management ..................................................................................... 30

2.5 Summary and Discussion ..................................................................................................... 34

Chapter 3 General Framework for Autonomic Workload Management........................................ 36

3.1 Admission Control Component ........................................................................................... 39

3.2 Scheduler ............................................................................................................................. 40

3.3 Execution Control Component ............................................................................................ 41

3.4 Workload Coordinator ......................................................................................................... 41

3.5 Summary .............................................................................................................................. 43

vi

Chapter 4 Discovering Indicators for Congestion .......................................................................... 45

4.1 Indicator Discovery .............................................................................................................. 48

4.1.1 Training Data Collection ............................................................................................... 48

4.1.2 Raw Data Pre-processing .............................................................................................. 51

4.1.3 Monitor Metric Selection .............................................................................................. 55

4.2 Experiments ......................................................................................................................... 57

4.2.1 Experimental Environment ........................................................................................... 58

4.2.2 Disk I/O Contention ...................................................................................................... 61

4.2.3 Shared Data Contention ................................................................................................ 66

4.2.4 Discussion ..................................................................................................................... 69

4.3 Summary .............................................................................................................................. 71

Chapter 5 Prototype of Autonomic Workload Management ......................................................... 73

5.1 Prototype System ................................................................................................................. 74

5.1.1 Scenarios and Policies ................................................................................................... 75

5.1.2 Prototype Implementation ............................................................................................. 76

5.1.3 Feedback Control Approach ......................................................................................... 84

5.2 Experiments ......................................................................................................................... 87

5.2.1 Experimental Environment ........................................................................................... 87

5.2.2 Congestion .................................................................................................................... 88

5.2.3 SLA Violation ............................................................................................................... 91

5.3 Summary .............................................................................................................................. 94

Chapter 6 Conclusions and Future Work ....................................................................................... 96

6.1 Conclusions .......................................................................................................................... 96

6.2 Future Work ....................................................................................................................... 100

References .................................................................................................................................... 102

Appendix A DB2 (v9.7) Monitoring Table Functions ................................................................. 113

Appendix B TPC-H Benchmark .................................................................................................. 136

Appendix C TPC-C Benchmark .................................................................................................. 139

vii

List of Figures

Figure 1.1 Performance Degradation Due to Workloads Competing for System Resources ........ 3

Figure 2.1 IBM‟s MAPE Loop of an Autonomic Manager ......................................................... 11

Figure 2.2 A Taxonomy of Workload Management Technique for DBMSs............................... 20

Figure 3.1 General Framework for Autonomic Workload Management ..................................... 37

Figure 3.2 MAPE Loop Implemented in the Framework ............................................................ 43

Figure 4.1 Experimental Framework for Training Data Collection ............................................. 49

Figure 4.2 Raw Data Pre-processing Steps .................................................................................. 51

Figure 4.3 Algorithm for Generating a Decision Tree [34] ......................................................... 56

Figure 4.4 The Experimental Environment .................................................................................. 58

Figure 4.5 An Example of Defining Service Classes and Workloads in DB2 ............................ 59

Figure 4.6 An Example of Calling DB2 Monitoring Table Functions ........................................ 60

Figure 4.7 Workloads Used in the Case of Disk I/O Contention ................................................. 62

Figure 4.8 Throughput of the System in the Case of Disk I/O Contention .................................. 63

Figure 4.9 CPU Resource Utilization in the Case of Disk I/O Contention .................................. 64

Figure 4.10 Disk Resource Utilization in the Case of Disk I/O Contention ................................ 65

Figure 4.11 Indicators for the Case of Disk I/O Contention ........................................................ 65

Figure 4.12 Workloads Used in the Case of Shared Data Contention ......................................... 67

Figure 4.13 Throughput of the System in the Case of Shared Data Contention .......................... 68

Figure 4.14 Indicators for the Case of Shared Data Contention .................................................. 69

Figure 5.1 An Example of Workload Management Policies ....................................................... 76

Figure 5.2 A Prototype System of Autonomic Workload Management ...................................... 77

Figure 5.3 An Example of Defining a Threshold in DB2 ............................................................ 78

Figure 5.4 Policy Rule Propositions for the Admission Control ................................................. 79

Figure 5.5 An Example of Altering a Threshold in DB2 ............................................................. 80

file:///C:\PhD_Writings\%5b2008-2013%5d%20Breadth%20Depth%20Thesis\PhD-Dissertation\Final%20Version\thesis.docm%23_Toc378699894







viii

Figure 5.6 Policy Rule Propositions for the Execution Control ................................................... 81

Figure 5.7 An Example of Using DB2 Parameters and Procedures ............................................ 82

Figure 5.8 A Feedback Control System for Workload Control ................................................... 84

Figure 5.9 A Black-box Model for DBMSs ................................................................................. 85

Figure 5.10 Throughput of the System in Congestion without Control ....................................... 89

Figure 5.11 Throughput of the System in Congestion with Control ............................................ 90

Figure 5.12 Workloads Used in the Scenario of SLA Violation .................................................. 92

Figure 5.13 Throughput of the System in SLA Violation without Control ................................. 93

Figure 5.14 Throughput of the System in SLA Violation with Control ...................................... 94

Figure 6.1 Database Schema for TPC-C Benchmark [92] ......................................................... 139



ix

List of Tables

Table 2.1 The Fundamental Properties of an Autonomic Computing System............................. 10

Table 2.2 The Three Types of Controls in a Workload Management Process ............................ 18

Table 2.3 Summary of the Approaches Used for Workload Admission Control ........................ 24

Table 2.4 Summary of the Approaches Used for Workload Execution Control.......................... 29

Table 4.1 Summary of the Two Cases of Congestion .................................................................. 70

Table 6.1 Information returned for MON_GET_WORKLOAD [45]........................................ 113

Table 6.2 Information returned for MON_GET_BUFFERPOOL [45] ..................................... 119

Table 6.3 Information returned for MON_GET_TABLESPACE [45]..................................... 123

Table 6.4 Information Returned for MON_GET_CONTAINER [45] ...................................... 128

Table 6.5 Information returned for MON_GET_SERVICE_SUBCLASS [45] ........................ 130

Table 6.6 Tables and Cardinalities in TPC-H Benchmark [92] ................................................. 136

Table 6.7 Queries in TPC-H Benchmark [92] ........................................................................... 137

x

List of Acronyms

BI Business Intelligence

DBAs Database Administrators

DBMSs Database Management Systems

DDL Data Definition Language

DML Data Manipulation Language

IP Internet Protocol

IT Information Technology

MAPE Loops Monitor–Analyse–Plan–Execute Loops

MPL Multi-Programming Level

OLTP On-line Transaction Processing

SLAs Service Level Agreements

SLOs Service Level Objectives

SQL Structured Query Language

TCP Transmission Control Protocol

TPC Transaction Processing Performance Council

TPC-C Transaction Processing Performance Council – Benchmark C

TPC-H Transaction Processing Performance Council – Benchmark H

TPS Transactions per Second

WL Workload

WLM Workload Management

1

Chapter 1

Introduction

In order to tackle the crisis arising from the complexity of managing IT infrastructures, the

concept of autonomic computing was introduced by IBM in 2001 [40]. Since then, a great deal of

effort has been put forth by researchers and engineers in both academia and industry to build

autonomic computing systems. Autonomic computing is a term used to describe a broad range of

standards, architecture, products, services and tools that enable computing systems to be self-

managing. An autonomic computing system, with the ability to self- configure, self-optimize, self-

protect, and self-heal, is a self-managing system that manages its behavior in accordance with

high-level objectives, such as performance, availability or security, specified by human system

administrators. The goal of an autonomic computing system is to dynamically achieve the system

objectives and to shift tasks such as system configuration, optimization, maintenance, and fault

management from human system administrators to technology [29] [46].

Database management systems (DBMSs), as a major component of a computing system

in many organizations, are growing rapidly in scale and complexity. A DBMS may have

hundreds of parameters which are tuned for optimal performance, and performance tuning on a

major component may lead to unanticipated effects on the entire system. As computing systems

become increasingly complex, business organizations require more and more skilled IT

professionals to install, configure, tune, and maintain their systems. It is estimated that one-third

to one-half of a company‟s total IT budget is spent on maintenance and development of existing

enterprise computing systems [29]. In addition, the emerging trend to server consolidation renders

the management of DBMSs even more complicated. This trend, for a database server, requires a

2

single server to simultaneously process multiple workloads with diverse characteristics, dynamic

resource demands and distinct levels of business importance while at the same time satisfying

service level objectives (SLOs) for all the workloads.

1.1 Motivation

As outlined above, for both strategic and financial reasons, businesses are consolidating

individual database servers onto a shared database server. As a result, multiple types of requests

are simultaneously present on a single database server. Request types may include on-line

transaction processing (OLTP), business intelligence (BI) and administrative utilities. OLTP

transactions are typically short and efficient, consume minimal system resources and complete in

sub-seconds. BI queries tend to be more complex and resource-intensive, and may require hours

to complete. Administrative utilities are different from regular database queries in that their

execution, such as statistics collection, data reorganization and backup/restore, is essential to

preserve the system‟s integrity and efficiency. However, these actions can consume a large

quantity of system resources.

Requests submitted by different users or generated by different applications may have

different levels of business importance, e.g., high business importance or low business

importance (or best effort), and distinct SLOs. These properties of requests are typically derived

from a service level agreement (SLA). The SLA is a contract between a service provider and

service consumers that normally specifies what service is to be provided, how it is supported,

performance, costs and responsibilities of the parties involved. SLOs, on the other hand, are

specific measurable characteristics of the SLA, such as response time, throughput, availability or

quality. In this context, a workload is a set of requests that have certain common characteristics,

such as application, source of request, type of query, business importance or SLOs. It executes in

3

an environment where more than one type of workload is mixed with diverse characteristics,

varying resource demands, multiple levels of business importance, and distinct SLOs.

When multiple workloads execute concurrently on a database server, they compete for

shared system resources inevitably, such as system CPU cycles, buffer pool memory space, disk

I/O and various queues in the database system. If a workload consumes system resources without

control, the other workloads running concurrently in the system may have to wait for the

workload to complete and release the used resources. As a consequence, the waiting workloads

may fail to meet their desired SLOs and the database server may suffer severe performance

degradation. System performance refers to the way that the database system behaves in response

to the workloads, and it is measured in terms of the system‟s response time, throughput and

resource utilization [16] [45].

Figure 1.1 Performance Degradation Due to Workloads Competing for System Resources

0

650

0 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

Transactional WL Only

Transactional WL + Ad Hoc Query WL

t1

4

As an example, Figure 1.1 illustrates the dramatic performance degradation in a database

system resulting from executing a bursty ad hoc query (BI) workload, while a transactional

(OLTP) workload is running against the database system. An ad hoc query workload is a set of

randomly generated and drill-down knowledge mining queries. (The details of the experimental

setup are described in Chapter 4.) The throughput of the system without the ad hoc query

workload was high and stable, and averaged around 528 transactions per second (tps). When the

ad hoc query workload arrived at the system to execute at time t1, the throughput of the system

dropped dramatically, fluctuating rapidly between low and high levels. Thus, it is necessary and

critical for DBMSs to provide workload management functionality.

Workload management is the discipline of effectively monitoring and controlling work

flow across computing systems. In particular, workload management for DBMSs is the process or

act of monitoring and controlling work (or requests) executing on a database system in order to

make efficient use of system resources in addition to achieving any performance objectives

assigned to that work [44]. Thus, the primary goals of workload management in a DBMS are: 1)

to maintain the DBMS running in an optimal state (i.e., neither under-loaded nor over-loaded), 2)

to ensure that all workloads meet their desired SLOs, and 3) to balance resource demands of

workloads running concurrently and maximize performance of the entire system. A challenge is

that workloads present on a database server are typically unpredictable, varying highly in arrival

rates, types, and levels of business importance. As a result, it becomes virtually impossible for

database administrators (DBAs) to manually adjust the database system configurations to

dynamically meet SLOs, specifically performance objectives, for workloads during runtime.

Therefore, to achieve the SLOs in the complex mixed workload environment, autonomic

5

workload management becomes crucial to dynamically monitor and control the flow of

workloads and manage their demands on system resources.

An autonomic computing system is a self-managing system that manages its behavior in

accordance with high-level objectives specified by human system administrators. Such systems

regulate and maintain themselves without human intervention to dynamically achieve the

specified objectives and reduce the complexity of system management. In particular, an

autonomic workload management system for DBMSs is a self-managing system that dynamically

manages workloads present on a database server in accordance with specified high-level policies,

such as a workload management policy.

1.2 Thesis Statement

We claim that autonomic computing principles can be used to improve the

effectiveness of workload management in DBMSs. In particular, they can be used

to protect the DBMS against performance degradation and SLO violation.

The broad goal of this research is to add self-optimization capabilities to DBMSs, in order to

make the systems dynamically achieve the specified SLOs, such as response times or throughput,

for all workloads without requiring any intervention of human system administrators.

Specifically, the objectives of this thesis are the following:

To develop a taxonomy of workload management techniques to evaluate and classify the

existing workload management facilities provided in today‟s major commercial DBMSs,

such as IBM DB2, Microsoft SQL Server and Teradata Data Warehouse, and techniques

proposed in the recent research literature;

6

To develop a general framework for autonomic workload management in DBMSs, which

applies autonomic computing principles and incorporates multiple workload controls,

such as workload admission and execution controls, to dynamically control workloads in

the workload execution process;

To develop an approach to identifying a small set of key monitor metrics of a DBMS that

can serve as indicators to indicate impending severe performance degradation in the

DBMS;

To implement a prototype system of the framework on top of a commercial DBMS to

experimentally demonstrate the effectiveness of the framework and proposed approaches.

1.3 Contributions

This research makes several contributions to advance the state of the art in workload management

for DBMSs and in the practice of autonomic computing. The first contribution of this thesis is a

taxonomy of workload management techniques for DBMSs, and a survey of workload

management facilities (or systems) provided in today‟s commercial DBMSs and techniques

proposed in the recent research literature. The taxonomy categorizes the existing workload

management techniques into four technique classes, namely workload characterization,

admission control, scheduling and execution control. Within each class, the techniques are further

divided into subclasses based on their distinct mechanisms. The survey summarizes the main

features of typical workload management facilities, e.g., IBM DB2 Workload Manager [16],

Teradata Active Management System [90] and Microsoft SQL Server Resource Governor [67],

and some typical workload management techniques presented in the research literature. By using

the taxonomy of workload management techniques, the workload management facilities and

techniques are evaluated and classified.

7

The second contribution of this work is a general framework for autonomic workload

management in DBMSs, which allows a DBMS to adapt to highly varied and frequently changing

workloads and to dynamically meet the desired performance objectives without requiring

intervention by human system administrators. The framework consists of four major components,

namely admission control, scheduling, execution control and a workload coordinator, and

implements the feedback loop suggested in autonomic computing principles. The framework

provides the ability to dynamically monitor the database system and running workloads, to plan

what workload controls are effective if a performance issue is detected in the system, and then to

exert the controls on the workloads to make the workloads reach their desired performance

objectives and protect the database systems against performance failure.

The third contribution of this work is a learning-based approach to identifying indicators

to detect the impending congestion in a DBMS, which causes severe degradation in system

performance. The approach proposes the use of DBMS monitor metrics to act as congestion

(system-performance-failure) indicators. DBMSs typically make a very large number of monitor

metrics available, and collecting and analyzing monitoring data can impose significant overhead

and cause high latency. Thus, data mining techniques are exploited in the approach to identify the

key monitor metrics that can best indicate the problems, as system-performance-failure

indicators, and use the indicators to dynamically detect congestion problems.

The fourth contribution of the work is a prototype implementation of the general

framework for autonomic workload management in DBMSs. The prototype system implements

the framework on top of IBM‟s DB2, a commercial DBMS, by leveraging the existing workload

management techniques provided by the DBMS, and correspondingly creating a set of controllers

for the workload management techniques. Based on a specified workload management policy,

8

specifically, a high-level workload business importance policy, the prototype system coordinates

the multiple workload management techniques to react to various performance issues that arise in

the database system without requiring intervention of human system administrators. The

prototype system is used to conduct experiments to validate the proposed framework and

approaches.

1.4 Outline

The remainder of this thesis is organized as follows:

Chapter 2 reviews the principles of autonomic computing, and presents our taxonomy of

workload management techniques for DBMSs.

Chapter 3 illustrates our general framework for autonomic workload management in

DBMSs.

Chapter 4 explains our approaches to identifying system-performance-failure indicators,

which are used to detect severe performance degradation in database systems.

Chapter 5 describes the prototype implementation of our framework using DB2 Workload

Manager, and demonstrates the effectiveness of our framework and the identified system-

performance-failure indicators.

Chapter 6 concludes this work and suggests future research.

9

Chapter 2

Literature Review

This study draws from two main areas of research, i.e., workload management for DBMSs and

autonomic computing. This chapter provides an overview of these two areas. First, we introduce

the concept of autonomic computing in Section 2.1, and then discuss the underlying principles of

workload management for today‟s DBMSs in Section 2.2. In Section 2.3, we present our

taxonomy of workload management techniques developed by surveying workload management

facilities implemented in commercial DBMSs and techniques proposed in the research literature.

In Section 2.4, we review the work presented in previous research for exploring autonomic

workload management for DBMSs. Finally, we summarize the contents of this chapter, based on

our evaluation of the progress made on workload management, and discuss open problems in

Section 2.5.

2.1 Autonomic Computing

In 2001, IBM suggested the concept of autonomic computing [40]. The initiative aims to provide

the foundation for computing systems to manage themselves in accordance with specified high-

level objectives (that is, by specifying what is desired, not how to accomplish it) without direct

human intervention. Thus, autonomic computing systems are capable of reducing the complexity

of managing the computing environment and infrastructure, and reducing the burden on the

human system administrators. By choosing the word autonomic, IBM makes an analogy with the

human autonomic nervous system which controls activities, such as heartbeat, blood pressure and

breathing that enable human body to self-regulate and adapt to changing conditions. In the same

way, autonomic computing systems anticipate system requirements and resolve problems without

10

human intervention. As a result, human system administrators can focus on tasks with higher

value to the business.

Table 2.1 The Fundamental Properties of an Autonomic Computing System

Property Description

Self-configuring An autonomic computing system configures itself according

to high-level objectives.

Self-optimizing An autonomic computing system optimizes its use of

resources.

Self-protecting An autonomic system protects itself from malicious attacks.

Self-healing An autonomic computing system detects and diagnoses

problems.

An autonomic computing system, i.e., a self-managing system, has four fundamental self-

managing properties, namely self-configuring, self-optimizing, self-healing, and self-protecting,

as listed in Table 2.1 [29] [38] [46]. Self-configuring means that computing systems are able to

automatically configure system components to adapt to dynamically changing environments. This

property allows the addition and removal of system components or resources without system

service disruptions. Self-optimizing means systems automatically monitor and tune the resources

to ensure optimal functioning with respect to the defined requirements. This capability results in

the initiation of a change to the system proactively in an attempt to improve performance or

quality of service. Self-protecting means computing systems are able to proactively identify and

protect themselves from arbitrary attacks. This property enables a computing system to recognize

and circumvent data, privacy and security threats, but also threats from end users who

inadvertently make software changes, for example, by deleting an important file, and facilitates

the system to protect itself from physical harm, such as excessive heat or motion. Self-healing

11

means systems are able to recognize and diagnose deviations from normal conditions and take

action to normalize them. This property enables a computing system to proactively circumvent

issues which could cause service disruptions.

To achieve these autonomic (self-managing) capabilities, IBM has suggested the use of

closed-loop processing, which is associated with the concept of a Monitor–Analyse–Plan–

Execute (MAPE) loop shown in Figure 2.1 [38] [43] [46]. In Figure 2.1, the Managed Element

represents any software or hardware resource, such as a database server, a wired network, or a

CPU, that is provided autonomic capabilities by coupling it with an Autonomic Manager. Sensors

collect information on the Managed Element. For a database server, collected information might

include the response time to client requests, the system throughput, or the server‟s CPU

utilization. Effectors carry out changes to the Managed Element, for example, system

configuration adjustments, such as changing values of configuration parameters in the database

server to adapt to new or changing workloads.

Monitor

Analyse Plan

Execute

Knowledge

Managed Element

Sensors Effectors

Autonomic Manager

Figure 2.1 IBM’s MAPE Loop of an Autonomic Manager

In an Autonomic Manager, the Monitor provides the mechanisms that collect, aggregate,

filter and report details collected from a Managed Element. The Analyzer provides the

12

mechanisms that correlate and model current situations. The mechanisms allow the Autonomic

Manager to learn about the environment and help predict future situations. The Planner provides

the mechanisms that construct the actions needed to achieve desired objectives. The planning

mechanism uses policy information (stored in a knowledge source) to guide its work. The

Executer provides the mechanisms that control the execution of a plan. The Knowledge

component consists of particular types of management data with architected syntax and

semantics, such as symptoms, policies, requests for change, and change plans. The Knowledge

can be stored in a knowledge source so that it can be shared among Autonomic Managers [38]

[43].

Figure 2.1 illustrates a closed-loop process [43]. That is, with some “symptoms” present

(e.g., performance issues arising in a database system), the symptoms can be used to help an

Autonomic Manager provide self-managing capabilities: the Monitor receives an event from one

or more Managed Elements or retrieves state data from the Managed Elements; the Analyzer

correlates and analyzes the events or state data to determine if a specific known situation exists;

the Planner determines how they should be processed if a situation is detected and there is a

known action or set of actions; the Executer processes the actions.

2.2 Workload Management

Research in workload management for DBMSs has been mainly driven by commercial database

vendors. As databases are becoming increasingly large and complex, providing features of

workload management in the DBMSs to help the systems to achieve desired performance

objectives has been a key factor for business success. By surveying the workload management

facilities provided in today‟s major commercial DBMSs and techniques proposed in the recent

13

research literature, we present background information on workload management technology for

DBMSs and introduce the underlying principles in this section.

In using workload management facilities of commercial DBMSs, e.g., IBM DB2

Workload Manager [16] [45], Teradata Active System Management [90] [91], Microsoft SQL

Server Resource and Query Governor [67] [68], Oracle Database Resource Manager [77] , HP

Neoview Workload Management Services [39] and Greenplum Database 4.2 [33], three main

steps are suggested to effectively manage the wide variety of work executing concurrently in a

database server:

Explicitly understand performance objectives (or goals) for all requests based on a formal

SLA (if any).

Clearly identify arriving requests in the database server.

Impose appropriate controls on the requests to manage their behaviors so that the requests

can make steady progress towards the desired performance objectives.

We discuss each of these principles of workload management in detail below.

2.2.1 Performance Objectives and Management Policies

As introduced in Chapter 1, a request executing in a database server may have an assigned

business priority (or a business-importance level) and performance objective, and both are

defined in terms of a SLO. The mapping from the business-level SLO to the specific business

priorities and performance objectives can be a non-functional process (that is, a function cannot

be defined for the mapping) that requires business mapping rules along with knowledge shared by

the DBAs. The performance objectives can be expressed by one or more performance metrics.

Typical performance metrics include response time, the elapsed time between the start

and completion of a request, throughput, the number of requests completed in a time unit, and

14

request execution velocity, the execution speed of a request in a database system [41]. Request

execution velocity can be simply described as the ratio of the expected execution time of a

request to the actual time the request spent in the system (the total time of execution and delay),

where the expected execution time can be obtained from historical observations in the system‟s

steady state. If an execution velocity is close to 1, the delay of the request is small, while an

execution velocity close to 0 indicates a significant delay. In particular, high priority requests,

such as the ones that directly generate revenue for business organizations, or those issued by a

CEO or VP of the organizations, may expect a quick response, and thus they need to complete

quickly. Low priority requests, such as the ones for generating routine business reports, can wait

longer to get results. In using request execution velocity, performance objectives as well as

business priorities of requests can be captured. That is, by checking if a request‟s execution

velocity is close to 1, it can be known that the request (no matter low or high priority) has met its

desired performance objective or not.

To efficiently manage end user‟s work, requests are grouped into workloads. As a result,

performance objectives of the requests can be expressed relative to a workload. In many

situations, workload performance objectives are described as averages or percentiles, such as the

average response time of transactions in an OLTP workload, or x % of queries in a workload for

data warehousing decision support system complete in y time units or less [16]. A workload with

a high business priority may be associated with a high performance requirement, or larger x and

smaller y values. For non-goal request workloads, there are typically no associated performance

objectives. However, their performance may be inferred from a system resource allocation plan,

such as “non-goal workloads may consume no more than r % of the total processor resources”

[16].

15

In a workload management process, workload management policies are important in

controlling the behavior of running requests, where policies are the plans of an organization to

achieve its objectives [58] [70]. Workload management policies may include determining if a

request can be admitted to a database system, how long the request has to wait in a queue for

available shared system resources, and how fast the request can be executed in the database

system. The policies are typically derived from the defined workload business priorities and

performance objectives, and the policies may be applied to all points in a workload management

process. At different points, policies may have different specifications and rules, such as

admission policies used for specifying how a request would be controlled during its admission to

the system, scheduling policies used for guiding the request scheduling processes of workload

management facilities, and execution control policies used for defining dynamic execution

control actions [52] [63] [103].

2.2.2 Request Identification

Having explicitly defined performance objectives and management policies to manage the end

user‟s work to achieve the desired performance objectives, current practice shows that arriving

requests need to be clearly identified when they present in a database server so that the requests

can be properly managed [16] [90]. Thus, various workload definition approaches are used to

identify the incoming requests.

The term workload is also used to refer to an object in today‟s commercial databases,

which is defined for monitoring and controlling a set of requests [16] [90]. Workload definition

approaches use classifications to map arriving requests to workloads. A defined workload can be

assigned a priority (at the business level) based on the SLA specifications when the workload is

defined. A business transaction processing workload, such as data insertions generated by

16

cashiers in a store, is always assigned high business priority as the transactions directly generate

revenue and should complete promptly. On the other hand, a business analysis workload, such as

a set of business report generation queries for the store i.e., a report-generation batch workload, is

often assigned a lower priority as report generation is a daily routine and may be done in any idle

time window during the day. High business priority workloads require high-priority access to

shared system resources so that their performance objectives can be reached and guaranteed.

Grouping requests into workloads simplifies the task of allocating resources and provides

visibility into how system resources are being used by each workload [16] [67] [91].

Assigning a request to a workload can be done based on the request‟s operational

properties, such as origin or type [16] [91], or by applying a set of user-defined criteria functions

[67]. A request‟s origin, which indicates “who” is making the request, can include properties such

as the application name, user name, application‟s session ID and client IP address. A request‟s

type specifies “what” the characteristics of the request are, such as types of statements, estimated

costs or estimated cardinalities. The types of request statements may include READ, WRITE,

Data Manipulation Language (DML), and Data Definition Language (DDL) [84]. Estimated costs

or cardinalities predict the consumption of various shared system resources. Criteria functions are

typically scalar functions that contain the logic to classify the incoming requests into workloads,

where the logic can be derived from request properties [67].

2.2.3 Workload Control

Current research proposes that a workload management process in DBMSs may involve three

different types of controls, namely admission, scheduling and execution control [50] [51], as

listed in Table 2.2, and the controls are guided by specified workload management policies [52]

[103]. Admission control determines whether or not newly arriving requests can be admitted into

17

a database system, thereby avoiding an increase in load while the system is busy. As introduced

in Chapter 1, requests have various execution times and resource demands. Admission control

identifies “problematic” requests, such as the long-running and resource-intensive requests, and

makes appropriate admission decisions. The decision is based on the estimated costs of the

arriving requests and the specified workload management policies (specifically admission control

policies). The cost is typically estimated by the database query optimizer [11] [30]. If a request‟s

estimated cost exceeds the pre-defined admission threshold, the request may be queued for later

admission or rejected with a returned message. The pre-defined admission thresholds are

determined based on the admission control policies. The thresholds can include the upper limits

for the estimated resource usage of a request, the estimated execution time of the request, and the

number of requests running concurrently (the multi-programming levels) in the database system.

Workloads with different priorities can be associated with different admission control policies,

and therefore have different sets of threshold values. A high priority workload usually has higher

(less restrictive) thresholds, so high priority requests can be guaranteed to be admitted into the

database system for execution.

Request scheduling determines the execution order of requests in batch workloads or

admitted requests in wait queues (e.g., priority queues), and decides when the requests can be sent

to the database engine for execution based on the workload management policies (specifically

scheduling policies). The challenge of request scheduling is to determine the optimal number of

requests with various characteristics, priorities, and resource demands that can run concurrently in

a database system while maintaining the system in an optimal state and meeting the SLOs for all

workloads. Traditionally, the multi-programming levels (MPLs), a database system‟s threshold

discussed earlier, are used to manage the system load. If the MPL value is too large, the system

18

can become over-utilized, while, if the MPL value is too low, the system may be under-utilized.

In both cases, system performance suffers. For the same database system, different types of

workloads have different optimal MPLs. Request scheduling aims to dynamically set MPLs for

each of the workloads to decide which and how many requests can be sent to the database to

execute concurrently based on a specified scheduling policy.

Table 2.2 The Three Types of Controls in a Workload Management Process

Control Type Description Control Point Associated Policy

Admission

Control

Determines whether or

not an arriving request

can be admitted into a

database system.

Upon arrival in the

database system.

Admission control

policies derived from

a workload

management policy

Scheduling

Determines the execution

order of requests in batch

workloads or in wait

queues.

Prior to sending

requests to the

database engine for

execution.

Scheduling policies

derived from a

workload

management policy

Execution

Control

Manages the execution of

running requests to

reduce their performance

impact on the other

requests running

concurrently.

During execution of

the requests.

Execution control

policies derived from

a workload

management policy

In contrast with admission control and scheduling which are applied to requests before

their execution, execution control is imposed on a request during run time. The main goal of

execution control is to dynamically manage a running request in order to limit its impact on other

running requests, e.g., by slowing down the request‟s execution speed and freeing up shared

system resources for use by the other requests. Since query costs estimated by the database query

19

optimizer may be inaccurate, long-running and resource-intensive queries may get the chance to

enter a system while the system is experiencing a high load. These “problematic” requests

compete with others for the limited available system resources and result in the requests obtaining

insufficient resources and missing their desired performance objectives. Execution control

manages the running of the problematic requests based on an execution control policy and

determines to what degree the control should be applied.

2.3 A Taxonomy of Workload Management Techniques

In the past decade, considerable progress has been made in workload management for DBMSs.

New features of workload management facilities have been implemented in commercial DBMSs

and new techniques have been proposed by researchers. However, the descriptions of the

facilities and techniques in publically available documentation are very different in terms of their

terminology, even though their primary goals are the same, namely to achieve a set of per-

workload SLOs in a complex mixed workload environment. To facilitate the study and

understand the state of the art of the current workload management technology for DBMSs, we

develop a taxonomy as shown in Figure 2.2 to categorize workload management techniques based

on the main features of the techniques. The purpose of the taxonomy is to:

Classify the typical workload management techniques proposed in the research literature

and used in workload management facilities provided in DBMSs.

Highlight the relative strengths and weakness of existing techniques and to point out

deficiencies in the current set of techniques.

Our taxonomy is developed based on the controls involved in a workload management

process discussed in Section 2.2.3 and the techniques currently suggested by commercial database

20

vendors and in the recent research literature. We categorize the workload management techniques

for DBMSs into four major classes, namely workload characterization, admission control,

scheduling and execution control. Within a class, the workload management techniques are

further divided into subclasses based on their distinct mechanisms. In this section, we discuss the

main features of typical techniques in each class. These features are used to classify a particular

workload management approach.

Workload Management Techniques

Admission Control Scheduling Execution Control

Threshold-based

Control

Queue

ManagementSuspension ReprioritizationCancellation

Workload

Characterization

Dynamic

Characterizaton

Throttling Suspend & Resume

Query

Restructuring

Static

Characterization

Prediction-based

Control

Figure 2.2 A Taxonomy of Workload Management Technique for DBMSs

2.3.1 Workload Characterization

Workload characterization is essential for a workload management process as it provides the

fundamental information about a workload to its controllers. Workload characterization can be

described as the process of identifying characteristic classes of a workload in the context of the

workload‟s properties, such as costs, resource demands, business priorities and/or performance

requirements. A business transaction processing workload, for instance, is often characterized as

having low cost, few resource demands, high business priority, and requiring good performance.

While, on the other hand, a business analysis workload can be characterized as having high cost,

21

large resource demands, low business priority, and requiring best-effort (implicit) performance

objectives. The workload definition introduced in Section 2.1.2 is considered as a process of

workload characterization as, when a workload is defined, it is also characterized with regard to

its assigned business priority, estimated costs and expected performance behavior.

We divide workload characterization techniques into two types, namely static

characterization and dynamic characterization, as shown in Figure 2.2. Static workload

characterization defines the workloads before requests arrive and allocates shared system

resources to the defined workloads. This type of technique is widely employed in workload

management facilities provided in commercial DBMSs [16] [67] [90]. The main features of the

techniques are the differentiation of arriving requests based on their operational properties

discussed in Section 2.2.2, the mapping of the requests to a workload, and the resource allocation

to the workloads for their execution. Resource allocation is typically done based on the priority

assigned to a workload, such as high, medium or low. A workload with higher business priority

would have a higher priority to access shared system resources. Once assigned to a priority level

(at the system level), a workload has the rights defined for the priority level to access shared

system resources. However, the priority may be dynamically changed during the workload

execution based on the workload‟s performance requirements and actual performance behavior

[16] as explained in the discussion of Execution Control in Section 2.3.4.

Dynamic workload characterization identifies the type of a workload when it is presented

on a database server (e.g., an online transaction processing or an online analytical processing

workload). A typical technique proposed in the research literature for workload classification is

machine-learning [25] [26]. In using this technique, the system learns the characteristics of

22

sample workloads running on a database server, builds a workload classifier and uses the

workload classifier to dynamically identify unknown arriving workloads on the database server.

2.3.2 Admission Control

Traditionally, admission control in OLTP systems ensures that the number of client connections

is kept below a threshold so that the resource contention level among concurrent requests is

controlled. In the system, if the number of requests increases, throughput of the system increases

up to some maximum. Beyond the maximum, it begins to decrease dramatically as the system

starts thrashing [9] [22] [37]. In particular, admission control in mixed workload environments

aims not only to avoid accepting more work than a database system can effectively process, but

also to allow arriving requests to achieve their desired performance objectives.

We divide workload admission control techniques into two types, namely threshold-

based admission control and prediction-based admission control, as shown in Figure 2.2.

Threshold-based techniques specify the upper limit of a threshold, such as a system parameter,

under which an arriving query can be admitted. This type of technique is widely used in workload

management facilities provided in commercial DBMSs [16] [67] [90]. As described in Section

2.2.3, thresholds are used for controlling a request‟s admission, and the typical thresholds used

are query cost and the number of concurrently running requests (MPLs). The query cost

thresholds dictate that if a newly arriving query has estimated costs greater than the thresholds,

then the query is rejected, otherwise it is admitted. The MPL threshold dictates if the number of

concurrently running requests reaches the threshold, then no new requests are admitted into the

system. Workloads may be associated with different sets of threshold values based on a specific

admission control policy. Requests with higher priorities can be admitted into the system for

23

execution. The admission control policy may also specify different thresholds for various

operating periods, for example during the day or at night.

As an alternative to using system parameters, researchers have proposed threshold-based

techniques that rely on performance or monitor metrics, such as the conflict ratio, the transaction

throughput in time intervals of the recent past, and system-performance-failure indicators. The

conflict ratio [69] is the ratio of the total number of locks that are held by all transactions in the

system and total number of locks held by active transactions. If the conflict ratio exceeds a

(critical) threshold, then new transactions are suspended, otherwise they are admitted. The

approach that uses the transaction throughput [35] is a feedback method. The approach measures

the transaction throughput over time intervals. If the throughput in the last measurement interval

has increased (compared to the interval before), more transactions are admitted; if the throughput

has decreased, fewer transactions are admitted. The indicator approach [102] [103] uses a set of

monitor metrics of a DBMS to detect the performance failure. If the indicator‟s values exceed

pre-defined thresholds, low priority requests are no longer admitted. The basic idea of each

approach is to monitor some metric or set of metrics that indicate the current degree of resource

contention in the system, and to react to changes based on specified admission control policies. A

summary of the threshold-based approaches used for workload admission control is shown in

Table 2.3.

Prediction-based techniques attempt to predict the performance behavior characteristics

of a query before the query begins running [28] [31]. These techniques build prediction models

for queries using machine-learning approaches. Ganapathi et al. [28] find correlations among the

query properties, which are available before a query‟s execution such as the query‟s SQL

statement or the query‟s execution plan produced by the query optimizer, and query performance

24

metrics such as elapsed time and disk I/O, and use these statistical relationships to predict the

performance of newly arriving queries that have the same properties. Gupta et al. [31] build a

decision tree based on a training set of queries, and then use the decision tree to predict ranges of

the new query‟s execution time. Apart from being applied in workload management, this type of

technique can also be applied in other areas, such as system capacity planning [94] [95].

Table 2.3 Summary of the Approaches Used for Workload Admission Control

Threshold Type Description

Query Cost

[16][67][90] System Parameter

If an arriving query‟s estimated cost is greater than the

threshold, the query‟s admission is denied, otherwise,

accepted.

MPLs

[16][67][90] System Parameter

If the number of concurrently running requests in a

database system has reached the threshold, an arriving

request‟s admission is denied, otherwise, accepted.

Conflict Ratio

[69]

Performance

Metric

If the conflict ratio of transactions in a database system

exceeds the threshold, new transactions are suspended,

otherwise, admitted.

Transaction

Throughput

[35]

Performance

Metric

If the system throughput in the last measurement

interval has increased, more transactions are admitted,

otherwise fewer transactions are admitted.

Indicators

[102][103] Monitor Metrics

If the actual values exceed the pre-defined thresholds,

low priority requests are suspend, otherwise they are

admitted.

2.3.3 Scheduling

As introduced in Section 2.2.3, scheduling techniques for workload management involve sending

requests to the database engine for execution in a particular order that ensures the requests meet

their desired performance objectives and also maintains the database system in a normal (or

25

optimal) state. (The details of the state discussion are presented in Chapter 4.) In today‟s

commercial DBMSs, workload management facilities do not support request scheduling although

thresholds, such as MPLs, may be used for implementing certain functionalities of request

scheduling. As described previously, MPLs can be employed to manage the load in a database

system and therefore maintain the system in a normal state. However, threshold-based control is a

static mechanism. In a dynamic environment, threshold-based scheduling can result in the

database server running in an under-loaded or over-loaded state and cause the requests to miss

their required performance objectives.

In contrast with manually setting thresholds to control the load on a database server,

current studies have proposed a set of scheduling approaches. In the scheduling class of our

taxonomy, the typical techniques presented in the recent research literature can be divided into

two types, namely queue management and query restructuring, as shown in Figure 2.2.

The main features of queue management techniques are the determination of execution

order of requests based on the properties of the requests, such as resource demands, priorities and

performance objectives, and a scheduling policy [1] [32] [63] [76] [86]. After passing through an

admission control (if any), requests are placed in a wait queue or classified into multiple wait

queues according to their performance objectives and/or business priorities (a batch workload

may also be treated as a queue, in which a set of requests wait). A scheduler then orders requests

from the wait queue(s). The typical approaches include using a linear programming based

scheduling algorithm [1] [2] to determine an execution order for all requests in a batch workload,

or evaluating the admitted requests queued in the wait queue(s) using a utility function [76] or a

rank function [32]. The algorithms and functions take the request‟s estimated costs and

performance objectives as well as business priorities as input and, based on the returned values,

26

the scheduler determines the execution order for the requests. Before the requests can be released

from the wait queue(s) to the database engine to execute, the total available system resources

need to be estimated in order to keep the system running in a normal state. Specifically, the total

costs of executing requests should not exceed the database system‟s currently acceptable cost

limits [74] [75]. In order to maintain this constraint, studies show that queuing network models

[49] [55] or a feedback controller [19] [36] in conjunction with analytical models may be applied

[76] [87] [88]. The models and the controllers attempt to dynamically predict the MPLs for each

type of being released requests and keep the system running in a normal state.

Query restructuring techniques [54] decompose a query into a set of small queries, thus

the individual queries each being smaller and less complex than the whole. In the context of

query scheduling in workload management for DBMSs, a set of decomposed queries can then be

put in a queue and scheduled individually. In releasing these queries for execution one after

another, no short queries will be stuck behind large queries and no large queries will be required

to wait in the queue for long periods of time. By restructuring the original query, the work is

executed, but with a lesser impact on the performance of the other requests running concurrently.

One approach to query restructuring involves decomposing a large query execution plan into a

series of sub-plans that follow a certain execution order to produce results equivalent to the

original query [66].

2.3.4 Execution Control

Execution control aims to lessen the impact of executing work on other requests that are running

concurrently. We divide the request execution control techniques into three types, namely query

reprioritization, query cancellation and request suspension. Request suspension techniques are

27

further divided as request throttling and query suspend-and-resume, as shown in Figure 2.2. The

main features of these types of techniques are discussed in this section.

Query reprioritization involves dynamically adjusting the priority of a query as it runs,

thereby resulting in resource reallocation [7] [16] to the query according to its new priority. A

query‟s priority adjustment can be dynamically made through adjusting the priority of its

workload (at the business level) during execution. Normally, the priority of an end user‟s query

determines the resource access priority of the query. That is, high priority queries have greater

access to shared system resources while low priority queries are given lower priorities to the

resource access. Priority aging is a typical reprioritization mechanism implemented in

commercial DBMSs [16] [45]. The approach dynamically changes the priority of shared system

resource access for a request as it runs. When the running request tries to access more rows than

its estimated row counts or executes longer than a certain allowed time period, the request‟s

service level will be dynamically degraded, such as from a high level to a medium level, thus

reducing the amount of resources that the request can access. Thresholds, such as execution time

or the number of returned rows, are incorporated in the approach to adjust the priority of a

running request. The events of the threshold violation trigger the adjustment of a request‟s

priority level, and, in turn, adjust the priority of shared system resource access for the request

when the request‟s actual performance behavior violates the thresholds.

From the research literature, an approach proposed for the query reprioritization is

resource allocation driven by a specified workload importance policy [62] [99]. In this approach,

certain amounts of shared system resources are dynamically allocated to competing workloads

according to the workload‟s business importance levels. High business importance workloads are

assigned more resources, while low business importance workloads are assigned fewer resources.

28

The amount of shared system resources assigned to a workload can be dynamically changed to

respond to changes in the workload importance level as it runs. To enforce the workload business

importance policy to resource allocation among competing workloads, utility functions [48] [93]

are used to guide the dynamic resource allocation processes, and economic concepts and models

[20] [27] are employed to potentially reduce the complexity of the resource allocation problem

[8] [73]. The approach shows that more shared system resources can be dynamically allocated to

higher business important workloads than the ones with lower business importance during run

time.

Query cancellation is widely used in workload management facilities of commercial

databases [16] [77] [90] to kill the process of a running query. When a running query is

terminated, the shared system resources used by the query are immediately released, and the

impact of the query on the performance of concurrently running requests is directly eliminated.

The terminated query may be queued and re-submitted to the system for later execution based on

a request execution control policy [52] [103]. Like thresholds incorporated in the query

reprioritization techniques to trigger the adjustment of a request‟s resource access priority, a

query cancellation procedure can be automatically invoked in the system when a query‟s running

time or consumed shared system resources exceeds threshold values.

Request suspension means slowing down a request‟s execution. In the query suspend-

and-resume subclass, the main features of the techniques [10] [15] include terminating a query

when it is running, storing the necessary intermediate results and restarting the query‟s execution

at a later time. When a query is suspended, the resources used by the query are released and the

impact of the query on the performance of the concurrently running requests is eliminated. The

suspended query can be resumed when the database system is less busy. The main feature of

29

request throttling techniques [79] [80] [81] [82] is the dynamic manipulation of a request‟s

process as it runs. Instead of terminating a running query and storing its intermediate results,

request throttling pauses a running request to slow down its execution, and therefore, frees up

some resources used by the request, such as CPU shares and disk I/O. The difference between the

throttling and suspend-and-resume types of request suspension techniques is that request

throttling pauses the running queries for a certain time, and query suspend-and-resume terminates

running queries and continues their execution at a later time. A summary of the approaches used

for workload execution control is shown in Table 2.4.

Table 2.4 Summary of the Approaches Used for Workload Execution Control

Approach Type Description

Priority Aging

[16][45] Reprioritization

Dynamically changes the priority of system

resource access for a request as it runs.

Policy Driven

Resource

Allocation

[62][99]

Reprioritization

Amounts of shared system resources are

dynamically allocated to concurrent workloads

according to the levels of the workload‟s business

importance.

Query Kill

[16][77][90] Cancellation Kills the process of a request as it runs.

Query Stop-and-

Restart [10][15]

Suspend &

Resume

Terminates a query when it is running, stores the

necessary intermediate results and restarts the

query‟s execution at a later time.

Request Throttling

[79] [81] [82] Throttling Pauses the process of a request as it runs.

Apart from using a query execution time threshold to trigger the actions of dynamically

controlling a query‟s execution, query progress indicators can assist the request execution control

30

techniques decide whether or not a running query should be controlled based on a specific

execution policy. A query progress indicator attempts to estimate how much work a running

query has completed and how long the query will require to finish. This problem has been

recently studied, and a set of techniques [12] [13] [56] [59] [60] [61] have been proposed in the

research literature. Progress indicators keep track of a running query and continuously estimate

the query‟s remaining execution time. The difference between the use of query execution time

thresholds and query progress indicators is that thresholds have to be manually set whereas query

progress indicators do not need human intervention and therefore can potentially automate

request execution control.

2.4 Autonomic Workload Management

Applying autonomic computing principles in DBMSs has been extensively studied in the past

decade [5] [14] [17] [57] [64] [96]. In particular, in the studies of autonomic workload

management for DBMSs, autonomic components (with certain self-managing capabilities

discussed in Section 2.1) have been developed and shown to be useful in their own right, although

a large-scale fully autonomic computing system has not yet been realized [47] [65]. In this

section, we review typical work presented in the previous research for building autonomic

workload management.

Parekh et al. [79] [80] propose an autonomic database utility throttling approach to

dynamically limit the impact of on-line database utilities on end users‟ work. Database utilities, as

introduced in Chapter 1, include statistics update, index rebuild, database backup and restore, and

data reorganization. These are essential operations for a database system‟s good performance as

well as operation and maintenance, but when they are executed on-line, the database utilities can

significantly degrade the performance of production applications (end user‟s work). The authors

31

attempt to show how these on-line utilities can be controlled so the performance degradation of

production applications can be managed at an acceptable level.

In their approach, a self-imposed sleep is used to slow down (throttle) the on-line utilities

by a configurable amount. In the work, the authors use the term throttling to refer to slowing

down the execution of utilities. All work present on a database system is divided into two classes,

namely utility and production application classes. The system monitors the performance of the

production applications and reacts according to high-level policies to decide when to throttle the

utilities and estimates the appropriate amount of throttling. The performance degradation of

production applications is determined by comparing the current performance with the baseline

performance acquired by the production applications. The authors assume a linear relationship

between the amount of throttling and system performance and use a Proportional-Integral

controller to control the amount of throttling, and a workload control function translates the

throttling level into a sleep fraction for the on-line utilities.

By extending the work of Parekh et al., Powley et al. [81] [82] propose an autonomic

query throttling approach to dynamically throttle large queries in a database system based on

high-level business objectives. Query throttling is a workload execution control approach that

slows the execution of large queries to free up resources for other work running concurrently in

the system to achieve the required performance objectives. In the approach, a self-imposed sleep

is used to slow down (throttle) long-running, resource-intensive and low-priority queries, and an

autonomic controller automatically determines the amount of throttling needed to allow other

concurrently running requests to meet their performance objectives.

Powley et al. use autonomic computing principles to build two different types of

controllers, a simple controller and a black-box model controller, and compare their effectiveness

32

in determining the appropriate amount of throttling to ensure that high priority workloads meet

their performance objectives. The simple controller is based on a diminishing step function, and

the black-box model controller uses a system feedback control approach [36]. The authors also

compare two query throttling methods, which are called constant throttle and interrupt throttle.

The constant throttling method involves many short pauses, which are consistent and evenly

distributed throughout a query‟s run time, thus slowing the query‟s execution. The interrupt

throttling method involves only one pause throughout a query‟s run time, and the pause length is

determined by the amount of throttling.

Niu et al. [74] [75] propose an autonomic query scheduler to manage the execution order

of multiple classes of queries in order to achieve the workloads‟ service level objectives (SLOs).

The query scheduler is built on an IBM DB2 database system [45] and uses DB2 Query Patroller

[42], a DB2 query management facility, to intercept arriving queries and acquire the query‟s

information. The Scheduler then dynamically determines an execution order of the queries and

releases the queries to the database engine for execution. The Query Scheduler works in two

processes, namely the workload detection process and the workload control process.

The workload detection process classifies arriving queries based on their SLOs, which

include the query‟s performance goal and business importance level, and monitors performance

to detect whether or not queries are meeting their performance goals. If the queries miss their

performance goals, the query scheduler calls the workload control process to change the mix rate

of the released queries in order to allow more important queries to meet their performance goals.

The query scheduler‟s workload control process implements a cost-based approach and

periodically generates new plans to respond to the changes in the mix of arriving requests. A

scheduling plan is generated based on the cost limits of the service classes to which arriving

33

queries belong, the arriving queries‟ performance goals and the database system‟s available

capacity. The cost limit of a service class is the allowable total cost of all concurrently running

queries belonging to the service class. The query scheduler uses utility functions to estimate how

effective a particular cost limit will be in achieving performance goals. An objective function,

which is defined based on the utility functions, is used to measure if a scheduling plan is

achieved, and an analytical model (a queuing network model [55]) is used to predict the system

performance when a scheduling plan is applied.

In the practice of autonomic computing, Kephart et al. [48] and Walsh et al. [93]

proposed the use of utility functions to achieve self-managing systems. In their work, the authors

presented utility functions as a general, principled and pragmatic way of representing and

managing high-level objectives to guide the behavior of an autonomic computing system.

Zhang et al. [100] [101] suggest that a utility function defined for database workload

management needs to be a monotonically non-decreasing function, and needs to be capable of

mapping the performance achieved by a workload with a certain amount of allocated resources

into a real number, u. In their work, the authors present a set of properties of a utility function in

autonomic workload management for DBMSs. That is,

The value, u, should follow the performance of a workload. Namely, it should increase or

decrease with the performance of the workload;

The amount of change in the utility should be proportional to the change in the

performance of a workload;

The input to a utility function should be the amount of resources allocated to a workload,

or a function of the resource allocation, and the output, u, should be a real number;

34

The value, u, should not increase (significantly) as more resources are allocated to a

workload, once the workload has reached its performance objective;

In allocating multiple resources to a workload, a utility function should capture the impact

of the allocation of a critical resource on performance of the workload.

Of the properties, the first two describe the general performance behavior of a workload

that a utility function needs to capture, and the third property presents the domain and codomain

of a defined utility function. The fourth and fifth properties represent the relationships among

workload performance, resource provisions, and performance objectives. The authors stated that

the properties can be used for evaluating whether or not an existing utility function is appropriate

for a specific system management scenario.

2.5 Summary and Discussion

In this chapter, we have introduced the concepts of autonomic computing, and discussed the key

properties of an autonomic computing system, which are self-configuring, self-optimizing, self-

healing, and self-protecting. To achieve the self-managing capabilities, we introduced the use of

closed-loop processing associated with the concept of a Monitor–Analyse–Plan–Execute (MAPE)

loop.

In understanding the state of the art of the current workload management technology for

DBMSs, we developed a taxonomy to categorize workload management techniques and illustrate

a workload management process. The workload management process includes arriving request‟s

classification, admission control, scheduling, and execution control. Each stage in the process is a

type of workload control, and it can be implemented with various techniques. The taxonomy can

35

also be used to identify the techniques employed in a workload management system and classify

a particular workload management approach into a technique class.

By surveying the workload management facilities provided in today‟s major commercial

DBMSs and techniques proposed in the recent research literature, we have found that the existing

facilities are still insufficient for use in mixed workload environments although they provide

mechanisms for workload management. As introduced earlier, workloads presenting on a today‟s

database server can vary dynamically and rapidly, whereas the workload management facilities

are typically “static”, such as thresholds used for controlling load on a database system. Thus, it is

difficult for DBAs to manually set the static thresholds to adapt to dynamic workloads and

achieve their performance objectives. While, the approaches proposed in the research literature

are mainly devoted to a particular aspect of database workload management, such as preventing

thrashing in a BI batch [63] or an OLTP [69] system, and they may not be applicable to the mixed

workload environments.

In the previous work on building autonomic workload management, despite all the efforts

to provide facilities to dynamically manage highly varied and frequently changing workloads, a

fully autonomic database workload management system has not yet been realized. In this study,

we use the existing workload management facilities in DBMSs, the multiple types of workload

controls, and the concept of MAPE loops to build an autonomic workload management system

for DBMSs.

36

Chapter 3

General Framework for Autonomic Workload Management

Despite the efforts of researchers and developers in both academia and industry to provide

facilities to effectively manage highly varied and frequently changing workloads, workload

management for DBMSs is still an open and hot research area [3] [4] [21] [53]. There are issues

that need to be explored and addressed in building today‟s workload management systems, which

include automatically choosing and applying appropriate workload management techniques to

manage end users‟ work during runtime, dynamically detecting impending severe performance

degradation in a database system, and reducing the complexity of a workload management

system‟s operation and maintenance.

Varied workload management techniques, as discussed in Chapter 2, have been

developed and implemented in most major commercial DBMSs. However, it is unclear what

techniques are appropriate and effective for a particular workload executing on the DBMSs under

certain circumstances, or how the multiple techniques can be coordinated to manage the

performance of all concurrently executing workloads and ensure that they meet their required

performance objectives. For example, consider a database server where there is an ad hoc query

workload (with low business importance) present while a large number of business-important

transactional requests are arriving. The execution of the ad hoc query workload may need to be

restricted in order to free up sufficient system resources for the important requests to reach their

desired performance objectives. To restrict the execution of the ad hoc query workload, workload

management techniques, such as query cancellation, throttling, and suspension & resumption,

may be applied. It is unclear which techniques are most appropriate and will be most effective

37

under the particular circumstances, and to what degrees the techniques need to be applied so that

the important requests meet their desired performance objectives with minimal penalty to the ad

hoc query workload. In addition, the interplay among multiple workload management techniques

can be difficult to anticipate at runtime.

To resolve these issues, we consider autonomic computing as the most effective approach

to implementing a workload management system. In particular, we apply autonomic computing

principles and exploit multiple types of controls described in our taxonomy of workload

management techniques to develop a general framework for autonomic workload management in

DBMSs to manage dynamic and mixed workloads in accordance with specified high-level

workload management policies.

Admission

ControlScheduling

Workload

Coordinator

Execution

Control

DBMS

DB

Mo

nito

r

DB

En

gin

e

Running

Requests

Requests

Workload

Management

Policies

Autonomic Workload Management

Requests Management Polices Messages

Results

Figure 3.1 General Framework for Autonomic Workload Management

38

Our general framework for autonomic workload management in DBMSs is shown in

Figure 3.1. It consists of four major components, namely admission control, scheduling,

execution control and a workload coordinator. As introduced in Section 2.2, the functionality of

the admission control component decides whether or not newly arriving requests can be admitted

into a database system; the scheduling component decides the execution order of the admitted

requests (or requests in a batch workload); and the execution control component decides when the

running process of a request needs to be controlled and to what degree. The workload coordinator

acts as the manager in the framework. It periodically accesses the database monitor to collect

information on system performance and the workloads, analyses the current state of the database

system and determines if the running workloads are meeting their performance objectives. If a

problem is detected, the workload coordinator messages to the workload control components

(i.e., admission control, scheduling and execution control) to initiate their controls, and to

coordinate their control actions. We assume that the workload management policies are defined

prior to the arrival of the requests shown in Figure 3.1. An example of workload management

policies used in our prototype implementation is presented in Chapter 5.

In Figure 3.1, the dark thick arrows represent the path of workload requests. The light

thin arrows denote 1) the message path among the four components (i.e., admission control,

scheduling, execution control, and workload coordinator), 2) the request and monitoring data

path between the workload coordinator and the database monitor, 3) the control path between the

execution control and the running requests in the database system, and 4) the output path to

clients. The dashed thin arrow denotes the distribution of the workload management policies

(which are also passed to the workload control components through the workload coordinator).

Workload management policies (e.g., workload business importance policies, which define the

39

level of business importance and performance objective of each workload in a mixed workload

environment) are typically derived from a SLA and are used to guide the behavior of the

workload control components.

3.1 Admission Control Component

The admission control component shown in Figure 3.1 consists of an admission controller and a

set of admission control mechanisms. The admission controller provides functionality to map the

high-level workload management policies to a low-level action policy of admission control. The

action policy specifies how system configurations are set in terms of the admission control, and

the low-level action policy is implemented with the admission control mechanisms. The typical

mechanisms are the threshold-based system parameters, or “tuning knobs”, such as multi-

programming levels (MPLs), as introduced in Section 2.3, that are provided by all major

commercial DBMSs. MPLs specify the upper limit of the number of requests that are allowed to

run concurrently in a database system. If the number of concurrent requests reaches the threshold,

the system rejects newly arriving requests being admitted. Different workloads can have different

threshold values specified in the action policy and the thresholds, e.g., MPLs, are set accordingly.

Requests with high business importance can therefore be guaranteed to be admitted into the

system for execution.

Threshold-based admission control relies on the characteristics or the estimated costs of

requests to make decisions, but it lacks knowledge of the request‟s actual performance behavior,

such as execution time or demands for each individual system resource, before they are admitted.

In OLTP systems, the threshold-based load control has been widely used and is effective because

workloads are “homogeneous”, that is, similar resource demands; while in mixed workload

environments some requests can be long-running and resource-intensive. When (even a few of)

40

the resource-intensive requests begin to run, they can significantly impair the performance of the

workloads, which are running concurrently, and cause severe performance degradation in the

system. Therefore, further controls such as scheduling and execution control, as discussed below,

become necessary.

3.2 Scheduler

The scheduling component shown in Figure 3.1 consists of a scheduler and one or more wait

queue(s), e.g., a priority queue. The queue is used for storing admitted requests prior to execution.

The scheduler provides functionality to determine the execution order for the requests. Namely,

the requests are chosen to execute using a scheduling algorithm. After passing through the

admission control, requests are placed in a queue or classified into multiple queues according to

the workload management policies, e.g., the requests‟ levels of business importance. The

scheduler then orders the requests from the queue. In the studies of request scheduling discussed

in Section 2.3, multiple approaches have been proposed, which include the use of linear

programming models [1], utility functions [76], and rank functions [32]. The approach using

linear programming models takes as input (i.e., constraints of the model) the request‟s estimated

execution times, costs and the request‟s performance objectives to determine an execution order

for the requests. The utility function approach defines utility functions for each of the queues (i.e.,

service classes presented in the study), and uses the defined utility functions to further define an

objective function. A solution to optimizing the objective function is a scheduling plan. The rank

function approach evaluates the requests using a rank function. The function takes the requests‟

estimated costs, execution times and performance objectives as input, and returns an optimization

plan or a value. Upon receiving the output, the scheduler determines the execution order for the

admitted requests. In these approaches, the estimated costs or execution times of requests are

41

obtained from the DBMS optimizer. To the best of our knowledge, no commercial DBMS

provides request scheduling features in their workload management facilities.

3.3 Execution Control Component

The execution control component shown in Figure 3.1 also involves a controller and a set of

control mechanisms. The execution controller provides functionality to map the high-level

workload management policies to a low-level action policy of execution control. The action

policy is carried out by the control mechanisms. Request cancellation and throttling [45], as

discussed in Section 2.3, are the two typical mechanisms provided by workload management

facilities in commercial DBMSs, such as IBM DB2 (v9.7) Workload Manager [16] [45]. Request

cancellation kills the running process of a request so that system resources used by the request are

released immediately, and the impact on the performance of the concurrently running requests is

eliminated directly. The killed request may be queued for later re-execution. Throttling slows

down the execution of a request‟s process as it runs (e.g., pauses the running request using self-

imposed sleep [80] [82]), thus freeing up some system resources, such as system CPU cycles and

disk I/O bandwidth, so the impact on other requests is reduced.

In workload execution control, the “good” targets are usually the unexpected “long-

running” and “resource-intensive” requests with less business importance. The action policies in

the framework aim to specify the target workloads, what execution control mechanisms need to

be used in a certain circumstance, and the amount of control required in order to achieve (at least

the important) workload performance objectives.

3.4 Workload Coordinator

The workload coordinator shown in Figure 3.1 includes a performance monitoring component, a

workload-performance analyzer and system-performance-failure indicators. The performance

42

monitoring component periodically collects performance information of the database system and

the workloads. The workload-performance analyzer compares the actual performance (e.g., the

monitored throughput) of running workloads to their desired performance objectives to detect

whether or not the performance objectives are being met. The system-performance-failure

indicators dynamically detect whether or not severe performance degradation (such as congestion

occurrence discussed in Chapter 4) is approaching in the database system.

In the framework, three types of workload controls (i.e., admission, scheduling, and

execution controls) are included, and all the components work in a coordinated manner. As an

example, if the system state information sent by the workload coordinator is congested, namely, a

severe system performance degradation has occurred, the admission control will suspend the

admission of any newly arriving “problematic” requests (such as the long-running and resource-

intensive ad hoc queries); the scheduler will not send any “problematic” requests to the database

engine for execution; and the execution control will initiate request throttling or invoke request

cancellation, to significantly limit or cancel the execution of the running “problematic” requests

in the database system. These coordinated workload controls help the system immediately

eliminate the performance problem and return to a normal state.

The framework implements the concept of a MAPE loop suggested in autonomic

computing principles, as shown in Figure 3.2. The database system and the currently executing

workloads are periodically monitored and the performance data is analyzed to determine whether

or not the workloads are meeting their desired performance objectives. If a problem is detected, a

plan is created that specifies which workload management techniques will be effective and the

plan is executed to exert the control on the workloads to ensure that the workloads reach the

desired performance objectives. This framework provides two levels of control on the workloads.

43

The workload coordinator determines when and which workload control component(s) needs to

be initiated, i.e., coordinates multiple workload control actions; and the initiated workload control

component(s) decides to what degree the controls need to be applied. The framework focuses on

controlling workloads at multiple stages, i.e., admission, scheduling and execution, in a workload

execution process, but not on dynamically allocating resources, such as CPU cycles or buffer pool

memory space to workloads. An approach to allocating multiple resources in DBMSs has been

proposed in our previous research [99].

Monitoring

(actual performance)

Analyzing

(objectives met?)

Planning

(workload controls)

Executing

(exert the controls)

Figure 3.2 MAPE Loop Implemented in the Framework

3.5 Summary

In this chapter, we present a general framework for autonomic workload management in DBMSs.

The framework consists of multiple workload management techniques, namely the admission

control, the scheduling and the execution control, and a workload coordinator that provides

dynamic performance monitor functions. The framework implements the concept of MAPE loops

suggested in autonomic computing principles, and provides the ability to help DBMSs adapt to

44

highly varied and frequently changing workloads, and dynamically meet desired performance

objectives without requiring any intervention of human system administrators.

As discussed in this chapter, the workload coordinator dynamically detects performance

issues arising in a database system. This functionality is implemented using a set of indicators. In

Chapter 4, we illustrate our approach to identifying a small set of key monitor metrics of a DBMS

that can serve as indicators to specifically indicate impending congestion in the DBMS. In

Chapter 5, we present a prototype system of the framework, which is implemented on top of IBM

DB2 (v9.7), a commercial DBMS, to experimentally demonstrate the effectiveness of the

framework and the proposed approaches.

45

Chapter 4

Discovering Indicators for Congestion

The workload coordinator, in our framework, provides the ability to dynamically detect

approaching severe performance degradation and initiates workload control activities to avoid the

performance problem. The functionality of detecting severe performance degradation can be

implemented using a set of system-performance-failure indicators. By monitoring the system-

performance-failure indicators, the workload coordinator detects impending performance failure

problems in a database system.

Severe performance degradation in a database system may occur for different reasons,

such as thrashing [22] and congestion [102]. System thrashing was initially reported in the first-

generation multi-programmed virtual memory computing systems. Designers expected system

throughput to increase toward a saturation level as the load increased. Instead, however, they

noticed that throughput began to significantly decline after a certain point (i.e., the “thrashing

tipping point”). Thrashing in operating systems is an unstable collapse of throughput of a system

as the load is increased, although the system does not crash. When thrashing occurs, the system

spends more time replacing data pages and less time actually processing them.

The solution to system thrashing is load control that manages the amount of load present

on a system. A well known solution is the working set model [24]. A request‟s working set is the

set of pages required to allow the request to make progress. If the request cannot be allocated its

minimum working set memory space, it is constantly queued at the paging disks to satisfy its

page faults. Thus, as long as requests are allocated sufficient memory space for the working set,

the system can be prevented from thrashing.

46

In particular, thrashing may occur in a database system for (at least) two distinct reasons,

namely memory contention or data contention [9] [69]. Memory contention is similar to the

virtual memory thrashing in operating systems. That is, excessive buffer replacement leading to

the over-use of disk I/O resources. This could be avoided if the working sets of the executing

requests are held in memory [6] [7] [63] [78]. Data contention involves excessive deadlocks or

lock conflicts leading to frequent and/or long blocking delays of requests in the system. This may

be avoided by the use of feedback control of workloads [9] [69], which periodically samples

some set of performance metrics. When the performance metrics exceed a certain threshold, the

rate of admitting requests to the system is reduced.

Thrashing behavior of DBMSs, as discussed above, has been intensively studied in

previous research. Thus, in our research, we focus on the problem of congestion. Congestion is a

performance issue that commonly occurs in mixed workload environments. For example, when

executing a mixture of business transaction processing and business analysis requests, the short

business transaction processing requests can become stuck behind the long business analysis ones

in the system. The short requests experience significant delay, and the database system suffers

severe performance degradation.

We define a DBMS as congested when the competition among requests for shared system

resources causes work in the system to stagnate and to not make any noticeable progress. The

distinction between thrashing and congestion is that thrashing occurs when database concurrency

increases to the “thrashing tipping point”, and congestion occurs when long requests hold

resources and cause database concurrency to decrease significantly.

As discussed above, performance failure problems, such as thrashing, have been studied

in previous studies, and a set of performance metrics have been proposed for indicating

47

performance deterioration in OLTP [69] and BI batch data warehousing systems [63]. However,

these indicators (i.e., the performance metrics) are applicable to the particular systems, and may

not be generic for all potential sources of system performance failure. We, therefore, develop a

generic approach that can be applied to all potential sources to identify potential indicators in a

specific DBMS.

We use internal DBMS monitor metrics to detect approaching system performance

failure problems, such as congestion. Typically, DBMSs make a very large number of monitor

metrics available (as an example, approximately seven hundred metrics were used in our study

discussed in Section 4.2) and collecting and analyzing monitoring data may impose significant

overhead and cause high latency. Thus, a data mining approach, namely a decision tree (as a

decision tree presents a set of key attributes) [34] [98], is used to identify the key monitor metrics,

(i.e., a small subset of monitor metrics that can best indicate congestion), and the discovered

indicators are used to dynamically detect the problems.

It is insufficient to use performance metrics of a database system, such as response time

or throughput, to indicate the impending problems as these performance metrics only indicate the

effects of severe performance degradation after the problem occurs in the system. Our objective,

however, is to detect the problems as they are approaching. Using physical resource utilizations

of the database server, such as CPU utilization, to signal the problems is also insufficient since a

database server may execute with other applications concurrently on the same computer. Thus, it

is difficult to identify whether or not severe performance degradation is occurring in the database

system when the computer‟s physical resource utilization reaches an undesired level.

This chapter is structured as follows. Section 4.1 describes our approach to discovering

congestion indicators in a DBMS. Section 4.2 presents experiments to illustrate congestion in

48

DBMSs leading to system performance deterioration and demonstrates the use of our approach to

derive the indicators. Section 4.3 summarizes the work.

4.1 Indicator Discovery

In our approach, we collect monitoring data from a DBMS in both its normal and transition

regions (described in Section 4.1.1), and use this monitoring data as a set of training data for the

system. The monitor metrics are then mined from the training data, extracting those which are

key to identifying that DBMS congestion is about to occur.

As discussed earlier, the main idea of our approach is to build a classifier that can classify

a DBMS‟s state in either a normal or a transition region. If a DBMS is detected in a transition

region, workload control is initiated to prevent the DBMS from further moving into a congested,

i.e., undesirable, region. The system state in a certain region is described using the values of the

internal DBMS monitor metrics, and the classifier is built off-line using a supervised learning

technique. In using the classifier, the DBMS state in a normal or a transition region can be

dynamically identified. Our approach consists of three main components, namely, training data

collection, raw data pre-processing, and monitor metric selection. Section 4.1.1 explains how a

set of training data can be experimentally collected, Section 4.1.2 demonstrates our method of

pre-processing the collected raw data, and Section 4.1.3 discusses the analysis of the pre-

processed data to discover indicators.

4.1.1 Training Data Collection

For a specific DBMS scenario (i.e., a configuration and a set of workloads), our approach divides

the DBMS state space, as shown in Figure 4.1, into three mutually exclusive regions: Normal,

where workloads reach their required performance objectives and/or the general system

performance is at desired levels; Transition, where the system begins to become inefficient and

49

suffer some degradation in performance; Congested, where running requests are blocked in the

system and not progressing, and the system is experiencing severe degradation in performance.

Requests Snapshots

Snapshot

Records

GroupCongested Region

Normal Region

DBMS State Space

Transition Region

Figure 4.1 Experimental Framework for Training Data Collection

Figure 4.1 illustrates our experimental framework for training data collection. In the

sample scenario, a set of training data is collected by running the DBMS under load. Initially, the

database server runs a light load, and its state is in the normal region. By incrementally adding

additional load onto it, the state of the server changes, conceptually moving from its normal

region to its transition region and eventually to its congested region. The database server is

sampled periodically by taking snapshots of the system running in all three regions. The samples

of the system, i.e., the system monitoring data, are saved in a database, or a set of text files, along

with a timestamp as they are taken. The number of samples is (practically) large, so the mean of

the monitor metrics can be estimated reasonably (discussed in Section 4.1.2) based on the central

limit theorem [71] and the state changes in the DBMS can be captured. After the experiment, the

samples are traced back with the timestamps, and a set of samples taken in the normal region and

the same number of samples taken in the transition region are selected. A class label, normal, is

attached to each sample taken from the normal region, and a class label, pre-congested, is

50

attached to each sample taken from the transition region. The samples are saved in a text file as a

set of training data. The training data is then fed into the raw data pre-processing component

(discussed in Section 4.1.2) for its pre-processing, and then sent to the monitor metric selection

component (discussed in Section 4.1.3) to produce a set of system-congestion indicators.

As introduced in Chapter 1, in a mixed workload environment, the concurrently running

workloads can have different types, business importance, performance objectives, as well as

arrival rates, and these properties can dynamically change during runtime. Therefore, it becomes

very difficult to discover system-congestion indicators by monitoring and analyzing the entire

system with its dynamic and mixed workloads. Thus, in emulating a mixed workload

environment in this study, we assume there is a workload, such as an OLTP workload, that runs at

a constant load (e.g., in a batch execution manner, or more generally, the workload can be

represented with a closed queueing network model [55]). By tracking the workload (as an OLTP

workload is often characterized as having predictable performance behavior and high business

importance, it becomes sensitive to any change during its execution), and collecting and analyzing

its monitoring data, namely analyzing the records of the workload execution behavior collected in

the samples, a set of key monitor metrics can be identified, where the values of the monitor

metrics would be able to present critical changes when the database system moves into its

transition region from its normal region.

The „chosen‟ workload can then be monitored using the discovered monitor metrics to

indicate whether or not the database system is on the verge of congestion whenever additional

requests are made. Thus, based on the assumption, instead of monitoring and sampling the entire

DBMS and its dynamic and mixed workloads, only the chosen workload with its relevant

database objects, such as buffer pools [84] [89], need to be monitored and sampled.

51

4.1.2 Raw Data Pre-processing

The collected DBMS monitoring data consists of a set of system samples. A monitoring data

record is comprised of values of a set of monitor metrics, such as the number of data logical

reads, the number of data physical reads, or the total amount of CPU time used by the system

used in IBM DB2 [45] (refer to Appendix A for details), which are included in the system

snapshots. In this study, we refer to each monitor metric as an attribute, and each row as a record

in the collected monitoring data set.

Typically, the values of monitor metrics are the system‟s accumulated counter values,

which are determined by the DBMS‟s specific environment, such as its configurations. Since the

accumulated counter values cannot be directly used for discovering the key monitor metrics, we

employ data pre-processing to eliminate the accumulative effects on the collected monitoring

data and then normalize the data.

calculating differences

calculating moving averages

calculating z-scores

discretization

Figure 4.2 Raw Data Pre-processing Steps

The raw data pre-processing includes four steps, namely, calculating differences,

calculating moving averages, calculating z-scores, and discretization, as shown in Figure 4.2.

The monitoring data set has (n+1) attributes (columns), namely (a1, ..., ak, ..., an, ch), where, n

52

N, ak is the k-th attribute, (1 ≤ k ≤ n), and ch is the class label attribute, for h = 1,2, and c1 =

normal and c2 = pre-congested. There are t samples (rows) in the monitoring data set, where, t/2

samples belong to the normal class, and the other t/2 samples belong to the pre-congested class,

and t 2N. We denote a monitoring data record as 𝑟𝑖 = (𝑎1𝑖 ,𝑎2

𝑖 , ⋯ , 𝑎𝑘𝑖 ,⋯ , 𝑎𝑛

𝑖 , 𝑐ℎ), where, 𝑟𝑖 is

the i-th sample in the data set. 𝑎𝑘𝑖 represents the value of the k-th attribute in the i-th sample,

where 1 ≤ i ≤ t.

The first step, calculating differences, involves finding the difference between

corresponding attributes (for each attribute, 𝑎𝑘) in two consecutive records. By calculating the

differences, the accumulative historic effects on the data records are eliminated. If the i-th record

in the monitoring data set is 𝑟𝑖 = (𝑎1𝑖 ,𝑎2

𝑖 , ⋯ , 𝑎𝑘𝑖 ,⋯ , 𝑎𝑛

𝑖 , 𝑐ℎ), and the (i+1)-th record is 𝑟𝑖+1 =

(𝑎1𝑖+1 ,𝑎2

𝑖+1 ,⋯ , 𝑎𝑘𝑖+1 ,⋯ , 𝑎𝑛

𝑖+1 , 𝑐ℎ), then the differences between the two consecutive records,

𝑟𝑖and 𝑟𝑖+1, are:

∆𝑘𝑖 = 𝑎𝑘

𝑖+1 − 𝑎𝑘𝑖 (4.1)

The new data record is represented by 𝑑𝑖 = (∆1𝑖 , ∆2

𝑖 ,⋯ , ∆𝑘𝑖 ,⋯ , ∆𝑛

𝑖 , 𝑐ℎ), where, 𝑑𝑖 is the i-

th data record in the new data record set, denoted as D, and 𝑑𝑖 ∈ 𝐷, and ∆𝑘𝑖 is the k-th attribute

value of 𝑑𝑖 . This step is applied to both the normal and pre-congested classes, respectively. Thus,

the number of data records (rows) in the calculated new set, D, is | D | = t-2, where (t-2)/2 records

belong to the normal class, and the other (t-2)/2 records belong to the pre-congested class.

In order to reduce the amount of variation present in the collected monitoring data set, a

moving average of order m [34] is applied on the set, D, where, m N and 1 < m ≤ (t-2)/2. The

moving average is also applied to both the normal and pre-congested classes, respectively. In the

data set, D, (for each attribute, ∆𝑘), if the i-th data record is 𝑑𝑖 = (∆1𝑖 ,⋯ , ∆𝑘

𝑖 ,⋯ , ∆𝑛𝑖 , 𝑐ℎ), the

53

(i+1)-th data record is 𝑑𝑖+1 = (∆1𝑖+1 ,⋯ , ∆𝑘

𝑖+1 ,⋯ , ∆𝑛𝑖+1 , 𝑐ℎ), and so on, till the (i+m)-th data

record, 𝑑𝑖+𝑚 = (∆1𝑖+𝑚 ,⋯ , ∆𝑘

𝑖+𝑚 ,⋯ , ∆𝑛𝑖+𝑚 , 𝑐ℎ). A moving average of order m for the m data

records is calculated:

𝑔𝑘𝑖 =

∆𝑘𝑖 + ∆𝑘

𝑖+1 + ⋯ + ∆𝑘𝑖+𝑚

𝑚 (4.2)

The new data record can be represented by 𝑣𝑖 = (𝑔1𝑖 , 𝑔2

𝑖 ,⋯ , 𝑔𝑘𝑖 ,⋯ , 𝑔𝑛

𝑖 , 𝑐ℎ), where, 𝑣𝑖 is

the i-th data record in the new data record set, denoted as V, and 𝑣𝑖 ∈ 𝑉, and 𝑔𝑘𝑖 is the value of the

k-th attribute of 𝑣𝑖 , which is an average of the corresponding attribute values of those m

consecutive data records in the data set, D. Thus, the number of data records in the new data set,

V, is | V | = (t-2)-2m+2, where (t-2)/2–m+1 data records belong to the normal class, and the other

(t-2)/2–m+1 data records belong to the pre-congested class.

The third step, calculating z-score normalization [34] [71], is to normalize the data

records in the set, V, to generalize the data from its specific experimental environment, and to

reveal the differences in the data records in the normal and pre-congested classes. As discussed

earlier, we consider a stable workload constantly running in a complex database server

environment. The collected monitoring data for the stable workload in the normal region is

(almost) invariant. That is, the value increment of a monitor metric in each sampling interval, i.e.,

∆𝑘𝑖 defined in Equation (4.1), is more likely to be its mean value, while, in the case of the

transition region, the variance from the mean is likely to be greater. By taking a z-score

normalization, the differences in workload execution behavior in its normal and pre-congested

classes can be discovered. In the data record set, V, if the i-th data record is

𝑣𝑖 = 𝑔1𝑖 , 𝑔2

𝑖 ,⋯ , 𝑔𝑘𝑖 ,⋯ , 𝑔𝑛

𝑖 , 𝑐ℎ , (for each attribute, 𝑔𝑘 ; its associated class label attribute remains

the same), then its z-score normalization can be taken using equation (4.3):

54

𝑠𝑘𝑖 = 𝑎𝑏𝑠

𝑔𝑘𝑖 −𝐴

𝜎 (4.3)

In the equation (4.3), 𝐴 and are the mean and standard deviation, respectively, of the

attribute, 𝑔𝑘 , in the normal region, which can be experimentally estimated, and abs( ) is the

absolute function. The normalized data record is represented by 𝑧𝑖 = (𝑠1𝑖 , 𝑠2

𝑖 ,⋯ , 𝑠𝑘𝑖 ,⋯ , 𝑠𝑛

𝑖 , 𝑐ℎ)

where, 𝑧𝑖 is the i-th data record in the new data record set, O, and 𝑧𝑖 ∈ 𝑂, and 𝑠𝑘𝑖 is the k-th

attribute value of 𝑧𝑖 . Thus, the number of data records in the new data set, O, is same as the

number of data records in set V, i.e., | O | = | V |, where (t-2)/2–m+1 data records belong to the

normal class, and the other (t-2)/2–m+1 data records belong to the pre-congested class.

The fourth step of the data pre-processing is to discretize the normalized data records in

the set, O. This minimizes the possible values of an attribute (a continuous variable, attribute, can

have an infinite number of values) in order to simplify the attribute selection process (discussed

in Section 4.1.3). One approach to discretizing data records is the use of a ceiling function [34].

In the data set, O, if the i-th record is 𝑧𝑖 = (𝑠1𝑖 , 𝑠2

𝑖 ,⋯ , 𝑠𝑘𝑖 ,⋯ , 𝑠𝑛

𝑖 , 𝑐ℎ) (for each attribute, 𝑠𝑘 ; the

associated class label attribute remains the same), then the discretization can be done by applying

equation (4.4):

𝑙𝑘𝑖 = 𝑠𝑘

𝑖 (4.4)

The new data record is represented by 𝑤𝑖 = (𝑙1𝑖 , 𝑙2

𝑖 , ⋯ , 𝑙𝑘𝑖 ,⋯ , 𝑙𝑛

𝑖 , 𝑐ℎ) , where, 𝑤𝑖 is the i-

th data record in the calculated new data record set, W, and 𝑤𝑖 ∈ 𝑊, and 𝑙𝑘𝑖 is the value of the k-

th attribute of 𝑤𝑖 . The size of the new set, W, is same as the size of the set O, i.e., | W | = | O |,

where (t-2)/2–m+1 data records belong to the normal class, and the other (t-2)/2–m+1 data

records belong to the pre-congested class.

55

Using a coarse granularity of the discretization may significantly simplify the attribute

selection process (discussed in Section 4.1.3), but it decreases the accuracy of discovered

classifiers. While, on the other hand, using a fine granularity of the discretization increases the

accuracy of discovered classifiers, but it also increases the number of operations in the attribute

selection process. In our study, we consider the mapping: [0, 0.5] -> 0.5, (0.5, 1] -> 1, (1, 1.5] ->

1.5, (1.5, 2] -> 2, (2, 2.5] -> 2.5, (2.5, 3] -> 3, (3, 3.5] -> 3.5, and (3.5, +∞) -> 4 as we

experimentally found that this mapping provides us sufficient accuracy and reduces the number

of operations of the attribute selection process. Thus, the calculated values are in the set {0.5, 1,

1.5, 2, 2.5, 3, 3.5, 4}, which are used in our experimental result analysis discussed in Section 4.2.

4.1.3 Monitor Metric Selection

The goal of the monitor metric selection process is to identify a set of attributes (i.e., monitor

metrics) to act as classifiers (i.e., indicators) so that given a system sample with an unknown class

label, by testing the monitor metric‟s values, the class label of the data sample can be determined.

In our study, the information gain measure (a measure of the goodness of split) [34] [98]

is applied to select the attributes. Information gain is widely used in data mining applications in

choosing test attributes for building a decision tree. An attribute with the highest information gain

is chosen as the test attribute, which minimizes the information needed to classify samples in a

partition and reflects the least randomness or “impurity” in the partition [34]. Figure 4.3 shows

the algorithm for inducing a decision tree using the information gain measure from the set of

training data [34] [98].

Given the pre-processed data set, W, with n+1 attributes and (t-2)-2m+2 records, the

significant attributes are chosen iteratively. We calculate the information gain of all the attributes,

and the attribute with the highest information gain is chosen as a significant (test) attribute. Then,

56

all the data records in the set, W, are partitioned with respect to all possible values of this chosen

attribute. The information gain measure is then repeated in each of the partitions, iteratively.

Input: training set, W, attributes, (a1, ..., ak, ..., an)

Output: a decision tree (i.e., a set of key attributes)

Algorithm: Generate_Decision_Tree (W, (a1, ..., ak, ..., an))

1. attribute-list (a1, ..., ak, ..., an);

2. create a node, d;

3. if samples in W are all of the same class then

4. return d as a leaf node labeled with the class;

5. if attribute-list is empty then

6. return d as a leaf node labeled with the most common class in samples; //majority voting

7. select the attribute, ak, among attribute-list with the highest information gain;

8. label d with ak;

9. for each known value, vj, of ak //partition all the samples in training set W

10. grow a branch from node d for the condition ak = vj;

11. let Sj be the set of samples in W for which ak = vj; //a partition

12. if Sj is empty then

13. attach a leaf labeled with the most common class in the samples in W;

14. else attach the node returned by Generate_Decision_Tree (Sj, (attribute-list - ak));

Figure 4.3 Algorithm for Generating a Decision Tree [34]

57

In Figure 4.3, the information gain of the attribute ak, where, 1≤ k ≤n, is calculated using

equation (4.5) [34]:

𝐺𝑎𝑖𝑛 𝑎𝑘 = 𝐼 𝑒1 , 𝑒2 − 𝐸 𝑎𝑘 (4.5)

where, e1 is the number of records with the class label c1 in W, and e2 is the number of records

with the class label c2 in W. Thus, 𝐼 𝑒1 , 𝑒2 can be determined using equation (4.6) [34]:

𝐼 𝑒1 , 𝑒2 = − 𝑝𝑖 log2(𝑝𝑖)2𝑖=1 (4.6)

where, 𝑝1 = (𝑒1/𝑏), 𝑝2 = (𝑒2/𝑏) and b is the number of records in W. 𝐸 𝑎𝑘 can be determined

using equation (4.7) [34]:

𝐸 𝑎𝑘 = 𝑥1,𝑗 + 𝑥2,𝑗

𝑏𝑢𝑗=1 𝐼 𝑥1,𝑗 , 𝑥2,𝑗 (4.7)

where, the attribute 𝑎𝑘 has u distinct values, and 𝑥1,𝑗 is the number of samples, whose class label is

c1, and value of the attribute, 𝑎𝑘 , is the attribute‟s j-th value. 𝑥2,𝑗 is the number of samples, whose

class label is c2, and value of the attribute, 𝑎𝑘 , is the attribute‟s j-th value. 𝐼 𝑥1,𝑗 , 𝑥2,𝑗 can be

determined using equation (4.8) [34]:

𝐼 𝑥1,𝑗 ,𝑥2,𝑗 = − 𝑝𝑖𝑗 log2(𝑝𝑖𝑗 )2𝑖=1 (4.8)

where, 𝑝1𝑗 = (𝑥1,𝑗 /𝑋𝑗 ), 𝑝2𝑗 = (𝑥2,𝑗 /𝑋𝑗 ) and 𝑋𝑗 is the number of samples, whose value of the

attribute, 𝑎𝑘 , is the attribute‟s j-th value, and 𝑥1,𝑗 and 𝑥2,𝑗 are defined above in equation (4.7).

4.2 Experiments

In this section we present a set of experiments that illustrate two different cases of congestion in a

database system and demonstrate the effectiveness of our approach to discovering DBMS

58

congestion indicators. Section 4.2.1 describes our experimental environment. Sections 4.2.2 and

4.2.3 present the two different cases of congestion. In Section 4.2.4 we discuss the experimental

results.

4.2.1 Experimental Environment

The experiments were conducted with IBM DB2 (Version 9.7) database software [45] running on

an Acer® server with the Windows

® Server 2008 Enterprise operating system. The database

server was equipped with four Intel® Core

TM 2 Quad processors, 8GB of RAM, and two disks

configured with one disk controller. Three DB2 clients running on three PCs sent multiple types

of requests to the database server to build a multi-user and mixed-workload environment. TPC

benchmark workloads and databases [92] (refer to Appendices B and C for details) were used in

the experiments. One client issued TPC-C transactions and the other two clients generated TPC-H

queries and other requests (i.e., “DB-Lock” requests discussed in Section 4.2.3), respectively. The

operating system‟s performance monitor was used for collecting data on CPU and disk utilization.

Figure 4.4 shows the experimental environment.

Figure 4.4 The Experimental Environment

59

Congestion in database systems was presented on the DB2 database server, by adding

additional work to the database server, until it moved from its normal region to its transition

region and then to its congested region, while we sampled the system in all three regions to

collect a set of monitoring data.

In the experiments, we considered a TPC-C workload as the constant load workload, as

discussed in Section 4.1.1, in the DB2 database server. Thus, the workload and its related data

objects, such as bufferpool, tablespace, container and service class, were sampled. Buffer pools,

table spaces, and containers are the data objects provided in most database systems [84] [89].

Workloads and Service Classes are the objects in DB2, which are defined for providing workload

management features. An example of defining a service class and a workload object in a DB2

database system is shown in Figure 4.5.

In Figure 4.5, the “CREATE SERVICE CLASS” statement creates a service class, named

TPCC_SC, and the “CREATE WORKLOAD” statement creates a workload, named TPCCWL, in the

database system. All requests issued by the client PC with IP („130.15.6.46‟) will be assigned to the

TPCCWL workload, and they will run in the TPCC_SC service class.

DB2 CREATE SERVICE CLASS TPCC_SC

DB2 CREATE WORKLOAD TPCCWL ADDRESS („130.15.6.46‟) SERVICE CLASS TPCC_SC

Figure 4.5 An Example of Defining Service Classes and Workloads in DB2

60

The objectives discussed above were sampled using the DB2 monitoring table functions

[45] to access in memory counter values of the monitor metrics. The returned values were

concatenated as a data record and appended to the training data set. Each data record consisted of

approximately seven-hundred metrics.

An example of calling the table functions using DB2 SQL statements is shown in Figure

4.6. The MON_GET_WORKLOAD table function returns monitor metrics for the workload,

TPCCWL, which is the constant load workload. The MON_GET_BUFFERPOOL table function

returns metrics for the buffer pool, IBMDEFAULTBP, which is associated with the workload,

TPCCWL. The MON_GET_TABLESPACE table function returns monitor metrics for the table

space, USERSPACE1, which organizes the TPC-C data set. The MON_GET_CONTAINER table

function returns metrics for the table space, USERSPACE1, containers, which contains the TPC-C

data set. The MON_GET_SERVICE_SUBCLASS table function returns monitor metrics for

subclasses in the service class, TPCC_SC, which provides the execution environment to the

DB2 SELECT * FROM TABLE(MON_GET_WORKLOAD(„TPCCWL‟, -2))

DB2 SELECT * FROM TABLE(MON_GET_BUFFERPOOL(„IBMDEFAULTBP‟, -2))

DB2 SELECT * FROM TABLE(MON_GET_TABLESPACE(„USERSPACE1‟, -2))

DB2 SELECT * FROM TABLE(MON_GET_CONTAINER(„USERSPACE1‟, -2))

DB2 SELECT * FROM TABLE(MON_GET_SERVICE_SUBCLASS(„TPCC_SC‟, „ ‟, -2))

Figure 4.6 An Example of Calling DB2 Monitoring Table Functions

61

workload, TPCCWL. The information returned for the monitoring table functions shown in Figure

4.6 are listed in Appendix A.

After collecting the monitoring data, each sample was attached a class label as discussed

in Section 4.1.1, and the raw data pre-processing and the data mining approaches were then used

to identify the most important monitor metrics to indicate the congestion.

4.2.2 Disk I/O Contention

In the experiment, our TPC-C and TPC-H data and index tables were created in a database and

resided on a single disk. The sizes of the TPC-C and TPC-H data sets were 2GB and 6GB,

respectively. To isolate the copies of the data in main memory for the two workloads, two buffer

pools were created; one 1GB buffer pool associated with the TPC-C data set and another one

1GB buffer pool associated with the TPC-H data set. Only read-only TPC-C transactions, i.e.,

order-status and stock-level [92], were used to reduce disk I/O accesses generated by the TPC-C

workload and to make resources, disk I/O and CPU, both become potentially critical in the

system. TPC-H workloads are highly disk I/O intensive, and regular TPC-C workloads update

databases frequently. When TPC-H and TPC-C workloads execute concurrently, disk I/O often

becomes the critical resource in the system. Thus, we used the read-only TPC-C workload in the

experiment to ensure that either the disk or the CPU could become the critical resource. The mix

rate of the read-only transactions in the TPC-C workload was 50% for each.

The database server was run in its normal region using the TPC-C workload issued by a

client PC with 100 connections. The workload was run for 900 seconds, at which point TPC-H

queries, which were issued by another client PC with 200 connections, arrived at the system,

namely introducing bursts of ad hoc queries to the system, and, thus, moving the system into its

62

transition region then into its congested region. These two workloads were run concurrently for

another 900 seconds. Figure 4.7 illustrates the load from workloads used in the experiment.

Figure 4.7 Workloads Used in the Case of Disk I/O Contention

Figure 4.8 shows the system throughput during the experiment. When the transactional

workload was running alone, the system throughput was high and stable, however, once the TPC-

H queries began to execute, the throughput of the system dropped and fluctuated rapidly between

low and high levels. This severe degradation in the performance did not mean thrashing occurred

in the system as there was neither memory contention (as two separate buffer pools were used)

nor data contention (as the workloads accessed two separate data sets). The bottleneck resource in

the system was disk I/O since all data resided on a single disk and only one disk controller was

used. The highly I/O intensive TPC-H queries competed with the TPC-C transactions for the disk

I/O resources, occupying most of the disk I/O bandwidth in its execution and thus, impairing the

0

50

100

150

200

250

300

350

0 300 600 900 1200 1500 1800

The

Nu

mb

er

of

Co

ncu

rre

nt

Re

qu

est

s

Seconds


Transactional WL + Ad Hoc Query WL

63

performance of the TPC-C transactions significantly and resulting in severe degradation in system

performance.

Figure 4.8 Throughput of the System in the Case of Disk I/O Contention

Figure 4.9 shows the system CPU resource utilization, the percentage of time that the

CPU was busy serving requests in the database system, as the two workloads were executed.

When the transactional workload was executed alone, the CPU was fully utilized, and after the

TPC-H queries started to execute, the CPU utilization declined dramatically. Figure 4.10 shows

the disk resource utilization, the reads/writes of data from/to the disk per time unit, as the two

workloads were executed. With the transactional workload alone, the disk utilization was low, but

as the load increased, disk usage increased significantly. The trend of the physical resource

utilizations show that with the small load, requests in the system were CPU-bound, while running

0

100

200

300

400

500

600

700

0 300 600 900 1200 1500 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

64

with the large load, requests in the system became disk-bound. By observing the experimental

results of the resource utilizations, we see that with the two mixed workloads, requests were

delayed in the system waiting on the disk I/O resource. The CPU became almost idle, and the

throughput of the system became significantly lower and unstable.

Figure 4.9 CPU Resource Utilization in the Case of Disk I/O Contention

The experimental results shown in Figures 4.8 - 4.10 clearly illustrate the database

system‟s state change, namely, moving from the normal region to the transition region and then

to the congested region as the two workloads ran on the system. The system was in its normal

region when the TPC-C workload was running alone, and it was in its transition region when the

bursts of TPC-H queries began until the first sudden and sharp drop of its throughput, and then it

was in its congested region when the throughput was critically low and unstable.

0

20

40

60

80

100

120

0 513 1275 1411 1648 1797

CP

U U

tiliz

atio

n (

%)

Time (seconds)

65

Figure 4.10 Disk Resource Utilization in the Case of Disk I/O Contention

Using the monitoring data collected in the normal and transition region, respectively, and

applying the approach discussed in Section 4.1.1, a set of system-congestion indicators is

discovered for the case of disk I/O contention, as shown in Figure 4.11.

TCPIP_RECV_WAIT_TIME

TRANSITION REGION

> 1 <= 1

TOTAL_WAIT_TIME

NORMAL REGION

<= 2.5> 2.5

TRANSITION REGION

Figure 4.11 Indicators for the Case of Disk I/O Contention

0

400

800

1200

1600

2000

0 513 1275 1411 1648 1797

The

Nu

mb

er

of

Dis

k IO

/ S

eco

nd

Time (seconds)

66

Figure 4.11 shows a pruned decision tree [34] [98]. Using this decision tree, if the pre-

processed value of the metric “TCPIP_RECV_WAIT_TIME” (further discussed in Section 4.2.4)

contained in a data sample with an unknown class label is greater than 1, the class label of the

sample is classified as pre-congested. Namely, the DBMS is currently running in its transition

region. Otherwise, we examine the “TOTAL_WAIT_TIME” attribute (further discussed in

Section 4.2.4). If its pre-processed value is less than or equal to 2.5, then the data sample belongs

to the normal class, namely, the DBMS is currently running in its normal region. Otherwise, it is

classified as pre-congested.

Instead of manually pre-processing the collected raw data and building the decision tree,

we implemented our raw data pre-processing approaches discussed in Section 4.1.2 in a JavaTM

program and used the program to produce the pre-processed training data, and fed the training

data into Weka [97], a suite of machine learning software, to generate the decision tree.

4.2.3 Shared Data Contention

Congestion may occur in a database system for different reasons. In Section 4.2.2, we have

shown a case of physical resource, i.e., disk I/O, contention that causes congestion in a database

system. In this section, we show another case of congestion, which is caused by shared data,

namely lock contention.

In this experiment, we used an 8GB TPC-C database running a TPC-C workload with all

5 types of transactions [92] along with a database application designed to introduce lock

contention. The application, named “DB-Lock”, attempted to lock all the TPC-C data tables

(exclusive locks) for updating of the table‟s contents. The database server was run in its normal

region using the TPC-C workload alone issued by a client PC with 100 connections. The

workload was run for 900 seconds, at which point DB-Lock began, issued by another client PC

67

with a single connection, moving the system into its transition region and eventually to its

congested region. The two types of requests then ran concurrently for another 900 seconds.

Figure 4.12 illustrates the workloads as follows:

Figure 4.12 Workloads Used in the Case of Shared Data Contention

Figure 4.13 shows the system throughput as the two types of requests were executed.

When the transactional workload was running alone, the system throughput was high and stable,

however, once the DB-Lock began to execute, the system throughput decreased dramatically. The

DB-Lock request competed with the TPC-C transactions for locks of the TPC-C data set since the

application tried to lock all the tables to update their contents. DB-Lock held the locks for a long

period of time thus making requests from TPC-C wait for the locks to become available, therefore

resulting in severe degradation in overall system performance. This experiment illustrates the

0

20

40

60

80

100

120

140

0 300 600 900 1200 1500 1800

The

Nu

mb

er

of

Co

ncu

rre

nt

Re

qu

est

s

Time (seconds)

Transactional WL + DB-Lock WL Transactional WL Only

68

state change of the database system, namely, moving from its normal region to transition region

and then to its congested region The system was in its normal region when the TPC-C workload

was running alone, and it was in its transition region when DB-Lock began to execute until it

grabbed all the locks of the TPC-C data tables, and then the system was in its congested region

when the throughput totally declined. As the experiment was used to present logical resource

contention between the workloads, the physical resource utilization was not reported.

Figure 4.13 Throughput of the System in the Case of Shared Data Contention

By selecting the monitoring data collected in the normal and transition region,

respectively, and applying the approach introduced in Section 4.1, a set of system-congestion

indicators is discovered for the case of shared data contention, as shown in Figure 4.14.

0

25

50

75

100

125

150

0 300 600 900 1200 1500 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

69

Figure 4.14 shows two pruned decision trees [34] [98] presenting two separate sets of

system-congestion indicators. When using the pre-processed training data to build a decision tree,

we found that there were two attributes that could perfectly divide all samples into their own

classes. Either of these attributes can potentially act as a classifier for the database server‟s state.

The set presented on the left indicates that if the “TOTAL_PHYSICAL_READS” (discussed in

Section 4.2.4) pre-processed value of a data sample is greater than 0.5, the system is currently

running in its transition region, otherwise, it is in its normal region. Similarly, the right one

shows that if the “POOL_DATA_P_READS” (discussed in Section 4.2.4) pre-processed value of

a data sample is greater than 0.5 then the data server is currently running in its transition region,

otherwise, it is normal.

TOTAL_PHYSICAL_READS

NORMAL REGION

<= 0.5> 0.5

TRANSITION REGION

POOL_DATA_P_READS

NORMAL REGION

<= 0.5> 0.5

TRANSITION REGION

Figure 4.14 Indicators for the Case of Shared Data Contention

4.2.4 Discussion

Weka [97] was used in these two cases for generating and pruning the decision trees, as well as

for doing the cross validation for validating the classification accuracy of the discovered

indicators. Our results of cross validation showed that the correctly classified instances were

70

greater than 96% for both cases. The indicators can also be validated with test data sets, and if the

accuracy is lower than a required level, they need to be rediscovered with new training data. In

our study, we assume that all monitor metrics are independent. We also find that the transition

region is a short period. Table 4.1 shows a summary of the two cases.

Table 4.1 Summary of the Two Cases of Congestion

Reason of

Congestion

Types of Mixed

Workloads

Critical

Resources

Effects of

Congestion Discovered Indicators

Disk I/O

Contention

Transactions and

Ad Hoc Queries

Disk I/O

Resources

Throughput

dropped

dramatically

“TCPIP_RECV_WAIT_TIME”

and “TOTAL_WAIT_TIME”

Shared Data

Contention

Transactions and

DB-Lock Requests Table Locks

Throughput

declined

completely

“TOTAL_PHYSICAL_READS”

or “POOL_DATA_P_READS”

Through observing the discovered indicators in these two cases, we note that the

indicators can vary depending upon the source of congestion, and there may not be a set of

generic indicators for all potential sources. For a given DBMS/workload scenario, to discover a

set of indicators for the DBMS, the first phase is to identify a workload with high business

importance and „stable‟ execution behavior. We let the workload run on the system, and add

additional loads on the DBMS until it is congested. By collecting monitoring data in the normal

and transition regions, and applying our approaches, a set of indicators can then be discovered

and used to protect the performance of the DBMS.

The effectiveness of our approach is illustrated through the experiments. In the two

different cases, the TPC-C workload execution behavior was captured by the indicators. In the

71

first case, the monitor metrics, “TCPIP_RECV_WAIT_TIME” and “TOTAL_WAIT_TIME”, became

significant as the TPC-C workload generated less disk accesses when the database system moved

into its transition region from its normal region. A change in “TCPIP_RECV_WAIT_TIME”

indicated that the system spent more time waiting for an incoming client request over a TCP/IP

connection than when in the normal region. This was due to the fact that current requests were

stuck in the system, and new requests could not enter the system before the current requests

completed (the workload was the constant load, and it was executed in a batch execution manner,

as discussed in Section 4.1). A change in “TOTAL_WAIT_TIME” indicated that a transactional

request spent longer waiting within the database server. This monitor metric further described the

congestion situation. In the second case, the monitor metric, “TOTAL_PHYSICAL_READS” or

“POOL_DATA_P_READS”, showed that in a transition region, the TPC-C workload‟s execution

rate in the database server became slow, and therefore the workload could not generate as many

physical reads as it did in its normal region.

4.3 Summary

In this chapter we present a systematic study of congestion in DBMSs and propose a learning-

based approach to discovering congestion indicators in the DBMSs. Our approach consists of

three main components, namely training data collection, raw data pre-processing, and monitor

metric selection.

We divide the DBMS state space into three mutually exclusive regions, namely normal,

transition, and congested. The training data collection demonstrates a way to collect system

monitoring data in all three regions for a specific DBMS scenario. The raw data pre-processing,

consisting of four functions, namely calculating differences, calculating moving averages,

calculating z-scores, and discretization, presents a method to eliminate DBMS accumulative

72

effects on the collected monitoring data and then normalize the data. The monitor metric

selection, exploiting a data mining approach, identifies a set of key monitor metrics on the pre-

processed monitoring data to act as congestion indicators. In using the indicators, given a system

sample with an unknown class label of a state regain, by testing the monitor metric‟s values, the

DBMS state in a normal or a transition region can be dynamically identified.

The discovered indicators are a set of internal DBMS monitor metrics to indicate

impending congestion of a database system. Through a set of experiments, we illustrate

congestion, being caused by disk I/O contention and shared data contention, in DBMSs and

demonstrate our approaches to deriving the indicators. Our experimental results show that the

indicators can vary depending upon the source of system congestion.

73

Chapter 5

Prototype of Autonomic Workload Management

In the previous two chapters, we presented a general framework for autonomic workload

management and a learning-based approach to identifying indicators for detecting severe

performance degradation, such as congestion, in DBMSs. In this chapter, we present a prototype

system, which implements the framework on top of IBM‟s DB2, a commercial DBMS, and

employs the discovered congestion indicators. The prototype system provides the ability to

monitor the performance of the database system with its workloads, and to react to detected

performance problems, such as congestion and SLA violation. Multiple workload management

techniques, such as admission and execution control discussed in Chapter 3, are used to react to

the problems. Our prototype system implements the framework on top of DB2 (version 9.7)

Workload Manager [45] by leveraging its existing workload management techniques, and

creating a set of controllers for the workload management techniques to adapt to dynamic and

mixed workloads. Two different approaches are presented in the prototype implementation to

coordinate the multiple workload management techniques based on a high-level workload

management policy, e.g., a workload business importance policy. The workload management

policy is used to guide the workload control behavior of the multiple workload management

techniques.

This chapter is structured as follows. Section 5.1 illustrates the prototype implementation

of the framework for autonomic workload management. Section 5.2 presents a set of experiments

that evaluate the effectiveness of the framework and the discovered congestion indicators using

the prototype system. Finally, we conclude the work in Section 5.3.

74

5.1 Prototype System

In most commercial DBMSs, workload management facilities are the add-on components, which

are external to the database system. The database systems expose configuration parameters,

objects, and stored procedures to the workload management facilities to manage workloads. For

example, DB2 exposes a system configuration parameter, UTIL_IMPACT_LIM, for throttling the

execution of administrative utilities. The function slows down the execution of running

administrative utilities, such as RUNSTATS on tables (statistics collection) [45], so the impact of

the utilities on the performance of other workloads running concurrently in the system is limited.

The degree of throttling can be set by the parameter‟s values.

An example of a DB2 database object is workload, which can be defined to provide

request classification function. DB2 identifies an arriving request based on the source of the

request, such as a client IP address, or the type of the request, such as types of statements or

estimated costs. If workloads are defined, DB2 maps the request to the appropriate workload.

Another example of a DB2 database object is threshold, such as the MPL threshold,

CONCURRENTWORKLOADACTIVITIES and CONCURRENTWORKLOADOCCURRENCES, which

can be exploited to enforce concurrency control on requests within a designated domain or scope,

such as databases, service super-classes or service sub-classes, or workloads [45]. An example of

stored procedures in DB2 is WLM_CANCEL_ACTIVITY, which can be used to kill running requests

in the database system.

We build our framework on top of these workload management facilities. As these

parameters, thresholds, and stored procedures do not automatically adapt to workload changes,

we create controllers for the facilities. The controllers make the “static” workload management

facilities dynamic in accordance with the given workload management policies. Therefore, the

75

prototype system provides the ability to dynamically achieve performance objectives of

workloads with diverse characteristics and different levels of business importance while

protecting DBMSs against severe performance degradation.

5.1.1 Scenarios and Policies

Two different scenarios involving performance problems, namely system congestion and SLA

violation are considered in this chapter. In Chapter 4, we define a DBMS as congested when

competition among requests for shared system resources causes work in the system to stagnate

and to not make any noticeable progress. We also divide the state space of a DBMS into multiple

regions, including normal, transition, and congested. In this study, we use the same terms and

definition of congestion and DBMS state regions, and use the term SLA violation to refer to

workloads failing to meet their required performance objectives such as a certain required

throughput. Based on the two different problem scenarios, the workload management policy, a

workload business importance policy, is illustrated in Figure 5.1.

The policy shown in Figure 5.1 suggests two distinct classes of workloads in the database

system, namely a high priority and a low priority class, based on the workload levels of business

importance and performance requirements. The policy also specifies how the system regulates

execution of the less important workload if a problem is detected in the system, so performance

requirements of the important workload can be met. The policy is expressed at a high level, i.e., at

a business level, which only states workload business importance levels, performance

requirements, and what workload management activities are desirable, but not how to accomplish

them at the system level. Our prototype system provides the ability to translate the policy from

the business level to the system level and enforce the policy to the workload classes.

76

5.1.2 Prototype Implementation

As discussed above, our framework is built on top of DB2 (version 9.7) Workload Manager, so

the framework must be adapted to use the particular workload management facilities. DB2

Workload Manager provides mechanisms for admission and execution control, but not scheduling

(in terms of the discussions presented in Section 3.2). Thus, the prototype implementation shown

in Figure 5.2 consists of admission control, execution control, and workload coordinator

components.

In Figure 5.2, the admission control component (which implements the admission control

functional component shown in Figure 3.1) is comprised of DB2‟s mechanism to set MPL

thresholds, and an admission controller. The execution control component (which implements the

execution control functional component shown in Figure 3.1) is comprised of DB2‟s mechanisms

to perform throttling & cancellation, and an execution controller. The workload coordinator

(which implements the workload coordinator shown in Figure 3.1) with the admission controller

Direct revenue generating requests are important with explicit

performance requirements; all other requests are less important with

best-effort performance requirements.

Admit all arriving requests if system performance is at desired levels;

restrict less important requests if a performance problem, such as SLA

violation or congestion, is detected in the system.

Figure 5.1 An Example of Workload Management Policies

77

and the execution controller are implemented in a JavaTM

program and use JDBC to connect to

the database system. As this prototype implementation does not provide additional scheduling

algorithms (i.e., without a scheduling component), the admitted requests in the system are then

scheduled by DB2‟s default scheduling mechanisms. That is, a request starts executing as long as

it is admitted into the database system.

Workload

Management

PoliciesAdmission Controller

MPL Thresholds

DB2 Components

Request Classification

DB Monitor

Performance Monitoring

Workload Coordinator

Admission Control

Throttling & Cancellation

Execution Controller

Execution Control

Workload Execution

Leaving RequestsArriving Requests

Figure 5.2 A Prototype System of Autonomic Workload Management

In Figure 5.2, the first control activated in the workload management process is the

arriving request classification (identification) as the requests are mixed together when they arrive

at the system. In implementing this function, the request classification function of DB2 Workload

Manager is employed. As discussed earlier in this section, DB2 Workload Manager‟s

classification is based on the source of the request or the type of the request, and maps the request

78

to a defined workload. (A workload is a database object defined for monitoring and controlling a

set of requests [45].) In this implementation, we use the functionality and classify mixed requests

based on their unique client‟s IP addresses. Thus, multiple workloads are defined in the database

system for multiple request classes, respectively. An example of defining a workload and

mapping requests to the workload using the client‟s IP address is shown in Figure 4.5.

In implementing the admission control shown in Figure 5.2, the MPL thresholds,

CONCURRENTWORKLOADOCCURRENCES and CONCURRENTWORKLOADACTIVITIES, are used

as the admission control mechanisms. As discussed earlier in this section, these thresholds can be

used to control the concurrent level of requests within a designated domain or scope. Hence, we

create a threshold for each individual workload to control the number of requests from each

workload admitted into the database system. Workloads with different business importance can

be assigned different MPL upper bounds, and requests with high business importance are then

guaranteed to be admitted into the system for execution. An example of creating a threshold using

CONCURRENTWORKLOADACTIVITIES is shown in Figure 5.3.

DB2 CREATE THRESHOLD MAXACTIVITIESINTPCHWL

FOR WORKLOAD TPCHWL ACTIVITIES

ENFORCEMENT WORKLOAD OCCURRENCE

WHEN CONCURRENTWORKLOADACTIVITIES > 100

STOP EXECUTION

Figure 5.3 An Example of Defining a Threshold in DB2

79

In Figure 5.3, using the “CREATE THRESHOLD” SQL statement, we create a threshold

named MAXACTIVITIESINTPCHWL. The threshold uses the concurrency control mechanisms

provided by DB2‟s CONCURRENTWORKLOADACTIVITIES to enforce that the maximum number

of activities (requests) within the workload, TPCHWL, will not exceed 100. Note that the

workload, TPCHWL, is defined prior to the creation of the threshold object.

In guiding the admission control behavior, based on the workload management policies

presented in Section 5.1.1, we assume that the high-level workload management policies can be

represented with a policy description language, PDL [58]. The language uses event-condition-action

rules to map a series of events that a system is able to monitor into a set of actions that can be

invoked by a PDL policy. An event-condition-action rule is then triggered when the event occurs,

and if the condition is true the action is executed. Using the policy description language, a

collection of event-condition-action rules (i.e., a set of policy rule propositions) are specified to

describe the actions required in the admission control, as shown in Figure 5.4.

SystemPerformanceIssue

causes RequestAdmissionControl (

“less important workloads”, “suspension” )

if ( SystemPerformanceIssue.Case = “congestion” )


causes RequestAdmissionControl (

“less important workloads”, “suspension” )

if ( SystemPerformanceIssue.Case = “SLA violation” )

Figure 5.4 Policy Rule Propositions for the Admission Control

80

In the event-condition-action rules shown above, the event being monitored is “system

performance issues”, and the symbol, SystemPerformanceIssue, is associated with the event. The

event is also associated with an attribute, Case, to represent a class of performance issues, namely

“congestion” and “SLA violation” in this study. The policy rule propositions (i.e., the event-

condition-action rules) specify that the admission control is initiated if either case arises in the

database system. When the action is executed, the admission of less important workloads is

suspended and only important workloads are admitted. In the prototype implementation, these

propositions are coded in the admission controller shown in Figure 5.2. Thus, the MPL thresholds

are dynamically set in these cases. An example of setting the threshold values dynamically

(altering on-line) is shown in Figure 5.5.

In Figure 5.5, the “ALTER THRESHOLD” SQL statement alters the threshold,

MAXACTIVITIESINTPCHWL, to enforce that the maximum number of activities within the

workload, TPCHWL, will be less than 1. Thus, the admission of the workload, TPCHWL, is

suspended. The SQL statement is embedded in the admission controller shown in Figure 5.2 to

act as the admission control action. It is executed when the condition is true, that is, when the

DB2 ALTER THRESHOLD MAXACTIVITIESINTPCHWL

WHEN CONCURRENTWORKLOADACTIVITIES > 1

STOP EXECUTION

Figure 5.5 An Example of Altering a Threshold in DB2

81

performance issues arise in the database system. Note that the threshold and the workload are the

same objects used in Figure 5.3.

In implementing the execution control shown in Figure 5.2, the DB2 system parameter,

UTIL_IMPACT_LIM, and the stored procedures, WLM_CANCEL_ACTIVITY, are employed as the

execution control mechanisms. As discussed earlier in this section, the configuration parameter,

UTIL_IMPACT_LIM, can be used to initiate the throttling function of administrative utilities and

set the degrees. The stored procedure, WLM_CANCEL_ACTIVITY, kills running requests in the

database system. Our prototype system uses the mechanisms to dynamically control the execution

of less important workloads, so their performance impact on other workloads running

concurrently in the system is eliminated or reduced to a minimum. We use the policy rule

propositions, as discussed above in the admission control, to represent the actions required in the

execution control, as shown in Figure 5.6.


causes RequestExecutionControl (

“less important workloads”, “cancellation” )

if ( SystemPerformanceIssue.Case = “congestion” )


causes RequestExecutionControl (

“less important workloads”, “throttling” )

if ( SystemPerformanceIssue.Case = “SLA violation” )

Figure 5.6 Policy Rule Propositions for the Execution Control

82

The two policy rule propositions shown in Figure 5.6 specify that the execution control is

initiated if either of the performance issues, i.e., congestion or SLA violation, arises in the

database system. When congestion is detected, the execution of less important workloads is

terminated. The action of cancellation is executed as the system experiences severe performance

degradation, so the “immediate” control becomes necessary to remove the problem, and only

important workloads are still allowed to run. When SLA violation is detected, the execution of

less important workloads is fully throttled to reduce their impact to a minimum. The action of

throttling is executed as severe performance degradation does not occur in the system, so the

“slight” control can be sufficient. In this prototype implementation, the policy rule propositions

are coded in the execution controller shown in Figure 5.2. Therefore, the values of the system

configuration parameter or the stored procedure can be dynamically set or invoked when either of

the performance issues arises. An example of setting the system parameter and invoking the

stored procedure dynamically (i.e., on-line) is shown in Figure 5.7.

In Figure 5.7, the “UPDATE DBM CFG” SQL statement sets the system configuration

parameter, UTIL_IMPACT_LIM, to 1. It means that all running administrative utilities in the

system are fully throttled, and their performance impact on the other requests running

DB2 UPDATE DBM CFG USING UTIL_IMPACT_LIM 1 IMMEDIATE

DB2 CALL WLM_CANCEL_ACTIVITY(1, 2, 3)

Figure 5.7 An Example of Using DB2 Parameters and Procedures

83

concurrently is limited to no more than 1 percent of their original impact. The keyword

“IMMEDIATE” means that the change is made right away, i.e., on-line. The “CALL

WLM_CANCEL_ACTIVITY” SQL statement calls the stored procedure, WLM_CANCEL_ACTIVITY,

to cancel a running activity (request) with application handle 1, unit of work ID 2 and activity ID

3. The DB2‟s table function, WLM_GET_WORKLOAD_OCCURRENCE_ACTIVITIES [45], can be

used to find the application handle, unit of work ID, and activity ID of an activity. These SQL

statements are embedded in the execution controller shown in Figure 5.2 to act as the execution

control actions. They are executed, respectively, according to the policy rule propositions listed in

Figure 5.6, when the performance issues arise in the database system.

In implementing the workload coordinator shown in Figure 5.2, DB2‟s monitoring table

functions, such as MON_ GET_WORKLOAD and MON_GET_SERVICE_SUBCLASS discussed in

Section 4.2.1, are employed in the Performance Monitoring component to periodically access the

in-memory counter values of database monitor metrics, so performance information on the

database system and the workloads are dynamically collected. In this prototype implementation,

performance objectives of the important workloads and the identified system-congestion

indicators discussed in Section 4.2.2, are coded in the workload coordinator shown in Figure 5.2.

The workload coordinator compares the measured values to the encoded performance objectives

and the thresholds of system-congestion indicators to detect whether or not the workloads meet

their performance requirements and the system is currently in a normal state. If a problem is

detected, the workload coordinator sends a message to the workload controllers, i.e., the

admission controller and the execution controller, to initiate their controls. The message passed

between the workload coordinator and the workload controllers are implemented through calling

methods in the Java program.

84

5.1.3 Feedback Control Approach

As discussed previously in Section 5.1.2, a set of policy rule propositions is used to guide the

behavior of the multiple workload management techniques, i.e., the admission and the execution

control, and to define the degrees of the controls applied in each of the system performance

issues. In a fully implemented autonomic workload control system, we foresee the use of a

feedback control system [19] [36], as shown in Figure 5.8, to dynamically determine the amount

of the admission and the execution control.

-

+Performance

ObjectivesController Workloads

Throttling

DB2Components

Error MPLs

Measured Performance

Figure 5.8 A Feedback Control System for Workload Control

In Figure 5.8, the admission control, that is, dynamically setting the MPLs for workloads,

acts as a major control and the execution control, that is, dynamically setting the degree of

throttling for workloads, acts as a minor control. The admission control is major because the

MPLs set the maximum number of requests in the database system that can execute concurrently,

so it controls the contention level on system resources, and therefore impact on the system

performance directly. The execution control is minor because the throttling only slows down the

execution of running requests. In throttling, some used resources, such as CPU cycles and disk

I/O bandwidth, are released while the resources, such as locks and memory space, are still held by

85

the throttled requests. Therefore, the impact of the execution control on system performance is

not always significant. Throttling can be set to one of three levels, namely, high, intermediate or

low. Once a throttling level is set, the MPLs are correspondingly predicted by the feedback

control system. (In Figure 5.8, the dashed arrow denotes that throttling is not included in the

feedback control system.) The MPLs and the throttling are applied to the less important

workloads to control their impact and to maintain the performance objectives for the important

workloads. The amounts of the controls are appropriate (or optimal) when the important

workloads meet their performance objectives, while at the same time the less important

workloads are not penalized more than necessary. So, the appropriate throttling level can be

determined and set if the MPLs are maximized. That is, the system admits as many less important

requests as possible, while the performance objectives of important workloads are maintained.

DBMS

(Target System)Amount of

Control

u(k)

Measured

Performance

y(k)

The Mix of

RequestsInput d

Figure 5.9 A Black-box Model for DBMSs

To predict the MPLs, a black-box model (a statistical model) can be used to implement

the feedback control system. The term black-box model is used as only the input and output of the

target system need to be known [18] [36]. The DBMS can be modeled with a black-box model as

shown in Figure 5.9. The control input of the target system, u(k), is the amount of admission

86

control (i.e., the MPLs) imposed on the less important workloads and the measured output, y(k),

is the actual performance (e.g., the monitored throughput) of the important workloads running in

the DBMS. The disturbance, d, is the database system‟s mixed workload, which consists of all

types of requests, running in the system.

We consider a first-order model [36] to build a general controller and employ a linear

difference equation to describe the relationship between the control input and the measured

output [36]. The linear difference equation is defined as:

𝑦 𝑘 + 1 = 𝑎𝑦 𝑘 + 𝑏𝑢(𝑘) (5.1)

where, y(k+1) is the predicted performance in the (k+1)-st time unit, y(k) is the measured

performance in the k-th time unit and u(k) is the amount of admission control (i.e., the MPLs) for

the target system in the k-th time unit. The model parameters, a and b, can be experimentally

determined with least-square regression [71], as shown in equations (5.2) and (5.3) [36].

For parameter a:

2231

5243

SSS

SSSSa

(5.2)

For parameter b:

2231

4251

SSS

SSSSb

(5.3)

where, S1, S2, S3, S4 and S5 are temporary variables which are defined in the following equations

from (5.4) through (5.9) [36], respectively.

For variable S1:

𝑆1 = 𝑦2 𝑘 𝐾𝑘=1 (5.4)

87

For variable S2:

𝑆2 = 𝑢 𝑘 𝐾𝑘=1 𝑦 𝑘 (5.5)

For variable S3:

𝑆3 = 𝑢2 𝑘 𝐾𝑘=1 (5.6)

For variable S4:

𝑆4 = 𝑦 𝑘 𝐾𝑘=1 𝑦 𝑘 + 1 (5.7)

For variable S5:

𝑆5 = 𝑢 𝑘 𝐾𝑘=1 𝑦 𝑘 + 1 (5.8)

where, K is the number of samples, and y(k+1) and y(k) are the measured output of the target

system in the (k+1)st and k-th time unit in experiments. Due to the fact that we do not have the

access to DB2 source code, we were unable to implement this approach in our current prototype.

5.2 Experiments

In this section, we present a set of experiments using the prototype system. The experiments

illustrate the two scenarios, i.e., congestion and SLA violation, and examine the effectiveness of

our approaches implemented in the prototype system. Section 5.2.1 describes the experimental

environments. Section 5.2.2 and 5.2.3 present the two scenarios, respectively.

5.2.1 Experimental Environment

The prototype system was implemented using DB2 (version 9.7) database software on an Acer®

server with the Windows® Server 2008 Enterprise operating system. The database server was

equipped with four Intel® Core

TM 2 Quad processors, 8GB of main memory, and two disks

configured with one disk controller. Two DB2 clients running on two PCs sent different types of

88

requests to the database server to build a multi-user and mixed-workload environment. TPC

benchmark workloads and databases [92] were used in the experiments. One client issued TPC-C

transactions and the other client generated TPC-H queries and administrative utilities, i.e., run-

statistics on the TPC-H data tables. In the experiments we used the administrative utilities as

pseudo large BI queries because they can represent the table-scan and heavy-read requests, and

also can be throttled using the mechanisms provided by DB2, as discussed in Section 5.1.

In the experiments, we considered the TPC-C workload as the important workload with

an explicit throughput objective, e.g., 500 transactions per second (tps), and the TPC-H workload

and the administrative utility workloads as the less important workloads with best-effort (or

without explicit) performance objectives. We assume that if the important workload executes

alone, its performance objective can be met, while the database server still has available capacity

to process additional work. Thus, the less important workloads are sent and executed

simultaneously in the system. In this situation, the important workload may miss its performance

objective because of the increase in competition for system resources.

5.2.2 Congestion

In this experiment congestion shown in Section 4.2.2 was presented in the database system. The

prototype system was then used to dynamically detect the problem and control the workloads, so

the DBMS was protected from performance failure.

Our TPC-C and the TPC-H data and index tables were created in a database and resided

on a single disk. The sizes of the TPC-C and TPC-H data sets were 2GB and 6GB, respectively.

To isolate the copies of the data in main memory, two buffer pools were created; one 1GB buffer

pool associated with the TPC-C data set and another one 1GB buffer pool associated with the

89

TPC-H data set. TPC-C or TPC-H requests issued by each individual connection were in a batch

execution manner.

The database system was initially run in its normal region using the TPC-C workload

issued by 100 connections. The workload was run for 900 seconds, at which point the TPC-H

workload issued by 200 connections arrived at the system, i.e., introducing a bursty ad hoc query

workload to the system, and thus causing congestion in the system. Then, the two workloads were

run concurrently for another 900 seconds. (The workloads are illustrated in Figure 4.9.) In the

experiment, the read-only TPC-C transactions, i.e., order-status and stock-level [92], were used to

reduce disk I/O accesses generated by the TPC-C workload and to make disk I/O and CPU both

become potentially critical resources in the system. The mix rate of the read-only transactions in

the TPC-C workload was 50% for each.

Figure 5.10 Throughput of the System in Congestion without Control

0

100

200

300

400

500

600

700

0 300 600 900 1200 1500 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

90

Figure 5.10 shows the system throughput as the two workloads were executed without

using the prototype system. When the TPC-C workload was running alone, i.e., the system was

working in a normal state, the throughput was high and stable, however, once the TPC-H

workload began to execute, i.e., congestion occurred in the system, the throughput dropped

dramatically, fluctuating rapidly between low and high levels.

Figure 5.11 Throughput of the System in Congestion with Control

Figure 5.11 shows the system throughput as the two workloads were executed with using

the prototype system. In the experiment, the workload coordinator dynamically detected the

congestion problem using the system-congestion indicators discussed in Section 4.2.2, and sent

the congestion message to the admission and execution controllers. Based on the low-level action

policies specified in Section 5.1.2, the admission control set the value of the MPL threshold

0

100

200

300

400

500

600

700

0 300 600 900 1200 1500 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

91

associated with the TPC-H workload to zero, and thus suspended the admission of the TPC-H

workload. The execution control invoked the cancellation procedure, and thus cancelled the

running TPC-H queries in the system. Therefore, the severe performance problem was

immediately eliminated, and the system returned to a normal state.

5.2.3 SLA Violation

In this experiment we show that the important workload, i.e., the TPC-C workload, missed its

performance objective due to the execution of RUNSTATS administrative utilities [45]. The

prototype system dynamically detected and reacted to the problem and allowed the important

workload meet its performance objective.

A copy of the TPC-C data set and ten copies of PARTSUPP tables defined in the TPC-H

data set were created in a database and resided on a single disk. The size of the TPC-C data set

was 2GB, and the size of a PARTSUPP table was 1.07GB, so the total size of the PARTSUPP

tables was around 11GB. The TPC-C data set was used for the execution of TPC-C workloads,

and the PARTSUPP tables were used for the execution of RUNSTATS (statistics collection)

administrative utilities [45] (Note that any data table could be used in place of the PARTSUPP

table). In this experiment we use RUNSTATS administrative utilities as pseudo large BI queries,

and throttle the administrative utilities using the mechanisms provided by DB2, as discussed in

Section 5.1.2. Since ten copies of the PARTSUPP table were created, ten RUNSTATS utilities

could be concurrently executed in the database system. Two buffer pools were created in the

database system, namely one 1GB buffer pool associated with the TPC-C data set and one 4GB

buffer pool associated with the PARTSUPP tables to isolate the copies of the data in main

memory.

92

The TPC-C workload was used in the experiment with all 5 types of transactions [92]. It

was issued by 50 connections for a period of 900 seconds, at which point the RUNSTATS

utilities, issued by 6 connections, arrived at the system. The RUNSTATS introduced a heavy-read

workload to the system, which occupied large portion of the disk I/O bandwidth and caused the

TPC-C workload to miss its performance objective. The performance objective of the TPC-C

workload was set as 85 transactions per second (tps). The two workloads were run concurrently

for another 900 seconds. The requests issued by each of the connection were in a batch execution

manner. The two workloads are illustrated in Figure 5.12.

Figure 5.12 Workloads Used in the Scenario of SLA Violation

Figure 5.13 shows the system throughput as the two workloads were run without using

the prototype system. When the TPC-C workload was running alone, its performance objective

0

10

20

30

40

50

60

70

0 300 600 900 1200 1500 1800

the

Nu

mb

er

of

Co

ncu

rre

nt

Re

qu

est

s

Time (seconds)


Transactional WL + RUNSTATS WL

93

was met, however, once the RUNSTATS workload began to execute, the performance objective

was missed.

Figure 5.13 Throughput of the System in SLA Violation without Control

Figure 5.14 shows the system throughput as the two workloads were executed while

using the prototype system. In the experiment, the workload coordinator dynamically detected

that the business importance workload, i.e., the TPC-C workload, missed its performance

objective, i.e., 85 (tps). It then sent the SLA Violation message to the admission and execution

controllers. Based on the low-level action policies specified in Section 5.1.2, the admission

control set the value of the MPL threshold associated with the RUNSTATS workload to zero, and

thus suspended the admission of the RUNSTATS requests. The execution control invoked the

throttling mechanisms, and throttled the running RUNSTATS administrative utilities in the

0

20

40

60

80

100

120

140

0 300 600 900 1200 1500 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

94

system. Therefore, the performance problem was removed from the system, and the high

importance workload, i.e., the TPC-C workload, reached its performance objective again.

Figure 5.14 Throughput of the System in SLA Violation with Control

5.3 Summary

In this chapter, we present a basic version of our prototype system for autonomic workload

management in DBMSs. The prototype system implements our framework on top of IBM DB2

(version 9.7) Workload Manager. It leverages the existing workload management facilities, and

creating a set of controllers for the workload management facilities. The prototype system

provides the ability to monitor the performance of a database system and its complex workloads,

and to react to detected problems. The problems include systems suffering severe performance

0

20

40

60

80

100

120

140

0 300 600 900 1200 1500 1800

Thro

ugh

pu

t (t

ps)

Time (seconds)

95

degradation, and workloads missing their desired performance objectives. The prototype system

helps a DBMS adapt to highly varied and frequently changing workloads, and dynamically meet

the desired performance objectives without human system administrator‟s intervention. In the

prototype implementation, two different approaches are presented to coordinate the multiple

workload management techniques in reacting to various performance issues based on a given

workload management policy. The two approaches are the use of the PDL policy description

language and the application of a feedback control loop. Since we do not have the access to the

source code of DB2 database software, only the policy description language approach is

implemented in our prototype system.

In using the prototype system, we experimentally illustrated the effectiveness of our

framework shown in Chapter 3 and the derived congestion indicators presented in Chapter 4. The

experimental results demonstrated that our approaches are able to achieve the performance

objectives of workloads with diverse characteristics, multiple levels of business importance, and

varying resource demands and to protect database systems against severe performance

deterioration.

96

Chapter 6

Conclusions and Future Work

In this thesis, we present a study of autonomic workload management for database management

systems. We conclude our work in this chapter by providing a summary of the previous chapters

and discussing our future directions.

6.1 Conclusions

The broad goal of our research is to add self-optimization capabilities to DBMSs, in order to

make the systems dynamically achieve SLOs for all workloads without requiring intervention of

human system administrators.

We present a taxonomy of workload management techniques for DBMSs, and a survey of

workload management facilities provided in today‟s major commercial DBMSs and techniques

proposed in the recent research literature. Our taxonomy categorizes workload management

techniques and illustrates a workload management process. The workload management process

includes request characterization, admission control, scheduling, and execution control. Each

stage in the process is a type of workload control, and it can be implemented with various

techniques. The taxonomy can also be used to identify the techniques employed in a workload

management system and classify a particular workload management technique into a class.

In surveying the workload management facilities provided in today‟s major commercial

DBMSs and techniques proposed in the recent research literature, we have found that the existing

facilities are insufficient for use in mixed workload environments as they do not automatically

adapt to highly varied and frequently changing workloads. Similarly, current research approaches

to workload management are also insufficient as they tend to be devoted to a certain aspect of

97

database workload management, such as preventing thrashing in a BI batch or an OLTP system.

We propose the use of the existing workload management facilities provided in DBMSs, the three

types of workload controls, namely the admission, scheduling, and execution control, and the

concept of MAPE loop suggested in autonomic computing principles to build an autonomic

workload management system for DBMSs.

In Chapter 3, we present a general framework for autonomic workload management in

DBMSs, which allows a DBMS to adapt to highly varied and frequently changing workloads and

to dynamically meet the desired performance objectives without requiring intervention of human

system administrators. The framework consists of four major components, namely admission

control, scheduling, execution control and a workload coordinator, and implements the MAPE

loop suggested in autonomic computing principles. The framework provides the ability to

dynamically monitor the database system and running workloads, to decide which workload

controls are effective if a performance issue is detected in the system, and to exert the controls on

the workloads to make the workloads reach their desired performance objectives and protect the

database systems against performance failure.

Severe performance degradation in a database system may happen for different reasons,

such as thrashing or congestion. The workload coordinator, in our framework, detects the

approaching system performance failure problems. This functionality is implemented using a set

of system-performance-failure indicators. By monitoring the indicators, the workload coordinator

detects the impending performance problems in a database system. In Chapter 4, we present a

systematic study of congestion in DBMSs and propose a learning-based approach to identifying

congestion indicators in the DBMSs.

98

The identified indicators are a set of internal DBMS monitor metrics to indicate

impending congestion in a database system. Our approach consists of three main components,

namely training data collection, raw data pre-processing, and monitor metric selection. The

training data collection demonstrates a way to collect system monitoring data for a specific

DBMS scenario. The raw data pre-processing presents a method to eliminate DBMS

accumulative effects on the collected monitoring data and to normalize the data. The monitor

metric selection, through exploiting a data mining approach, identifies a set of key monitor

metrics on the pre-processed monitoring data to act as indicators.

In our study of congestion, we divide the DBMS state space into three mutually exclusive

regions, namely normal, transition, and congested. By applying our approach, we have obtained a

set of congestion indicators. Using the indicators, we dynamically detect approaching congestion

in a database system. Through a set of experiments, we illustrate congestion being caused by disk

I/O contention and shared data contention in database systems. Our results show that the

indicators can vary depending upon the source of congestion.

To experimentally illustrate and validate the framework for autonomic workload

management and the derived congestion indicators, we implement the framework on top of IBM

DB2 and use our derived congestion indicators. In Chapter 5, we present a basic version of our

prototype system. It leverages DB2 Workload Manager‟s existing workload management

facilities, and creates a set of controllers to implement a dynamic workload control system. The

system provides the ability to monitor the performance of a database system and its complex

workloads, and react to detected problems, such as congestion and SLA violation. The system

helps a DBMS adapt to highly varied and frequently changing workloads, and dynamically meet

the desired performance objectives. In the prototype implementation, two different approaches,

99

namely policy rule propositions and feedback control approaches, are discussed to coordinate the

multiple workload management techniques in reacting to various performance issues based on a

given workload management policy.

Using the prototype system, we experimentally illustrate the effectiveness our framework

shown in Chapter 3 and the derived congestion indicators presented in Chapter 4. The

experimental results demonstrate that our approaches are able to achieve the performance

objectives of workloads with diverse characteristics, multiple levels of business importance, and

varying resource demands and protect database systems against severe performance deterioration.

In conclusion, our study makes four main contributions in workload management for

DBMSs and in the practice of autonomic computing. The first contribution of this thesis is a

taxonomy of workload management techniques in DBMSs, and a survey of workload

management facilities provided by commercial DBMSs and techniques proposed in the research

literature. The second contribution is a general framework for autonomic workload management

in DBMSs. It consists of multiple workload management techniques and performance monitor

functions, and helps a DBMS adapt to dynamic and mixed workloads. The third contribution of

this work is a learning-based approach to identifying indicators to detect impending severe

performance degradation in a DBMS. The fourth contribution of the work is a prototype

implementation of the general framework for autonomic workload management in DBMSs.

These contributions advance the state of the art of workload management for DBMSs and the

practice in autonomic computing, and potentially reduce the burden from DBAs and the

operational costs of service providers.

100

6.2 Future Work

In implementing the framework for autonomic workload management, there are many issues that

need to be addressed. One of them is to relax the assumption that we made in the learning-based

approach to identifying indicators for detecting severe performance degradation in DBMSs. In the

current approach, we assume there is a constant load with high business importance running in

the mixed workload environment, so only the constant load and its relevant database objects are

monitored. In the next step of our research, we plan to investigate whether or not it is possible to

relax the assumption and to find indicators by monitoring and analyzing the entire system with its

dynamic and mixed workloads.

Another one is to extend the basic version of the prototype system to a fully implemented

version. Our current prototype system includes the admission control, execution control and

workload coordinator components but not scheduling. As a next step, we will integrate a

scheduling algorithm in the prototype system. There are several possible approaches [1] [32] [74]

that might be used. One possible approach is to implement the „utility function‟ approach [74] in

the prototype system to schedule requests based on their assigned business importance and

desired performance objectives. We also plan to apply feedback control techniques [18] to

translate the high-level workload management policies into low-level actions in terms of

workload controls, such as admission and execution control. As we do not have the access to DB2

source code, we plan to implement our proposed framework and approaches with the associated

workload management techniques and performance monitoring functions in an open source

database system, such as PostgreSQL [83] or MySQL [72].

Continuing our current studies, SLA-based Resource Optimization and Workload

Management is an interesting direction. Typically, SLAs are expressed in the contract between a

101

service provider and service consumers. The revenue of a service provider is the payments made

by service consumers, which may change with potential reduction in payments or even penalties,

based on the quality of the delivered services. The operational cost of a service provider is the

cost of resources used to run services. In our current research, the focus has been to explore how

the SLAs of consumer workloads can be satisfied dynamically, but less focus on improving

resource utilization to maximize service provider profit.

In the next step of our research, we plan to investigate SLA-based resource optimization

and workload management. We will explore whether or not it is possible to add capabilities to

DBMSs, which help the systems not only satisfy the SLAs for all competing workloads but also

maximize the system resource utilization. In order to reach this goal, multiple modeling

approaches may be applied. It may include using probability models [85] to model arrivals rates

and service times of workloads, using queuing models [49] [55] to model performance of

database systems and the workloads, and using Markov models [49] [85] to predicate the

occurrence of severe performance degradation in a database system. This research will advance

the state of the art of performance management for DBMSs, and can also be extended in building

infrastructures in cloud computing environments.

102

References

[1] M. Ahmad, A. Aboulnaga, S. Babu and K. Munagala. “Interaction-aware Scheduling of

Report-generation Workloads”. The International Journal on Very Large Data Bases.

Volume 20, Issue 4, August 2011. pp. 589-615.

[2] M. Ahmad, A. Aboulnaga, S. Babu, and K. Munagala. “Modeling and Exploiting Query

Interactions in Database Systems”. In Proc. of the 17th Intl. Conf. on Information and

Knowledge Management (CIKM’08). Napa Valley, CA, USA.

[3] A. Aboulnaga and S. Babu. “Workload Management for Big Data Analytics”. Tutorial at

the ACM Intl. Conf. on Management of Data (SIGMOD’13), New York, USA, June 2013.

[4] S. Babu, G. Graefe and H. Kuno. “Database Workload Management”. Dagstuhl Seminar

12282. Dagstuhl Reports, Volume 2, Issue 7, July 8-13, 2012.

[5] P. Belknap, B. Dageville, K. Dias and K. Yagoub. “Self-Tuning for SQL Performance in

Oracle Database 11g”. In Proc. of 25th Intl. Conf. on Data Engineering (ICDE’09). March

29 - April 2, 2009, Shanghai, China. pp. 1694-1700.

[6] K. P. Brown, M. J. Carey, and M. Livny. “Managing Memory to Meet Multiclass

Workload Response Time Goals”. In Proc. of the 19th Intl. Conf. on Very Large Data Bases

(VLDB’93). Dublin, Ireland. August 24-27, 1993. pp. 328-341.

[7] K. P. Brown, M. Mehta, M. J. Carey, and M. Livny. “Towards Automated Performance

Tuning for Complex Workloads”. In Proc. of the 20th Intl. Conf. on Very Large Data Bases

(VLDB’94). Santiago de Chile, Chile. September 12-15, 1994. pp. 72-84.

[8] H. Boughton, P. Martin, W. Powley, and R. Horman. “Workload Class Importance Policy

in Autonomic Database Management Systems”. In Proc. of 7th Intl. Workshop on Policies

for Distributed Systems and Networks, London, ON, Canada. June 5-7, 2006. pp. 13-22.

[9] M. J. Carey, S. Krishnamurthi, M. Livny. “Load Control for Locking: The Half-and-Half

Approach”. In (PODS’90): Proc. of the 9th ACM SIGACT-SIGMOD-SIGART Symposium

on Principles of Database Systems. 1990. pp. 72-84.

103

[10] B. Chandramouli, C. N. Bond, S. Babu, and J. Yang. “Query Suspend and Resume”. In

Proc. of the ACM SIGMOD Intl. Conf. on Management of Data. Beijing, China. June 11-

14, 2007. pp. 557-568.

[11] S. Chaudhuri. “An Overview of Query Optimization in Relational Systems”. In

(PODS’98): Proc. of the 17th ACM SIGACT-SIGMOD-SIGART Symposium on Principles

of Database Systems. 1998. pp. 34-43.

[12] S. Chaudhuri, V. Narasayya, and R. Ramamurthy. “Estimating Progress of Execution for

SQL Queries”. In Proc. of the 2004 ACM SIGMOD Intl. Conf. on Management of Data.

Paris, France. June 13-18, 2004. pp. 803-814.

[13] S. Chaudhuri, R. Kaushik, and R. Ramamurthy. “When Can We Trust Progress Estimators

for SQL Queries?” In Proc. of the 2005 ACM SIGMOD Intl. Conf. on Management of

Data. Baltimore, Maryland, USA. June 14-16, 2005. pp. 575-586.

[14] S. Chaudhuri and V. Narasayya. “Self-Tuning Database Systems: A Decade of Progress”.

In Proc. of the 33rd

Intl. Conf. on Very Large Data Bases (VLDB’07). Austria. pp. 3-14.

[15] S. Chaudhuri, R. Kaushik, A. Pol, and R. Ramamurthy. “Stop-and-Restart Style Execution

for Long Running Decision Support Queries”. In Proc. of the 33rd

Intl. Conf. on Very

Large Data Bases (VLDB’07). Austria, September 23-27, 2007. pp. 735-745.

[16] W. J. Chen, B. Comeau, T. Ichikawa, S. S. Kumar, M. Miskimen, H. T. Morgan, L. Pay,

and T. Väättänen. “DB2 Workload Manager for Linux, Unix, and Windows”. IBM

RedBooks, May 2008.

[17] B. Dageville, D. Das, K. Dias, K. Yagoub, M. Zait and M. Ziauddin. “Automatic SQL

Tuning in Oracle 10g”. In Proc. of the 30th Intl. Conf. on Very Large Data Bases

(VLDB’04). Toronto, Canada. 29 August - 3 September, 2004. pp. 1098-1109.

[18] Y. Diao, N. Gandhi, J. L. Hellerstein, S. Parekh and D. M. Tilbury. “Using MIMO

Feedback Control to Enforce Policies for Interrelated Metrics with Application to the

Apache Web Server”. In Proc. of 8th IEEE/IFIP Network Operations and Management

Symposium (NOMS’02), Florence, Italy, April 15-19, 2002. pp. 219-234.

104

[19] Y. Diao, J. L. Hellerstein, S. Parekh, R. Griffith, G. E. Kaiser and D. B. Phung. “A Control

Theory Foundation for Self-Managing Computing Systems”. IEEE Journal on Selected

Areas in Communications. Volume 23, Number 1, January 2005. pp. 2213-2222.

[20] D. L. Davison, G. Graefe. “Dynamic Resource Brokering for Multi-User Query

Execution”. In ACM SIGMOD Record, Volume 24, Issue 2, May 1995. pp. 281-292.

[21] U. Dayal, H. Kuno, J. L. Wiener, K. Wilkinson, A. Ganapathi and S. Krompass.

“Managing Operational Business Intelligence Workloads”. In ACM SIGOPS Operating

Systems Review, Volume 43, Issue 1, January 2009. pp. 92-98.

[22] P. J. Denning. “Thrashing: Its Causes and Prevention”. In Proceedings of the AFIPS Joint

Computer Conferences. San Francisco, California, USA. December 1968. pp. 915-922.

[23] P. J. Denning. “The Working Set Model for Program Behavior”. Communications of the

ACM, Volume 11, Issue 5, May 1968. pp. 323-333.

[24] P. J. Denning. “Working Sets Past and Present”. IEEE Transactions on Software

Engineering, Volume 6, Issue 1, January 1980. pp. 64-84.

[25] S. Elnaffar, P. Martin, and R. Horman. “Automatically Classifying Database Workloads”.

In Proc. of the 11th Intl. Conf. on Information and Knowledge Management (CIKM’02).

McLean, VA, USA. November 4-9, 2002. pp. 622-624.

[26] S. Elnaffar, P. Martin, B. Schiefer and S. Lightstone. “Is it DSS or OLTP: automatically

identifying DBMS workloads”, Journal of Intelligent Information Systems. Kluwer

Academic Publishers, Hingham, MA, USA. Volume 30, Issue 3, June 2008, pp. 249-271.

[27] D. F. Ferguson, C. Nikolaou, J. Sairamesh, and Y. Yemini. “Economic Models for

Allocating Resources in Computer Systems”. In Market-based Control: A Paradigm for

Distributed Resource Allocation. World Scientific Publishing Co., Inc. River Edge, NJ,

USA, 1996. pp. 156-183.

[28] A. Ganapathi, H. Kuno, U. Dayal, J. Wiener, A. Fox, M. Jordan and D. Patterson.

“Predicting Multiple Metrics for Queries: Better Decisions Enabled by Machine Learning”.

In Proc. of the 25th Intl. Conf. on Data Engineering (ICDE’09). March 29 - April 2, 2009.

Shanghai, China. pp. 592-603.

105

[29] A. G. Ganek and T. A. Corbi. “The Dawning of the Autonomic Computing Era”. IBM

Systems Journal, Volume 42, Issue 1, January 2003. pp. 5-18.

[30] G. Graefe. “Query Evaluation Techniques for Large Databases”. ACM Computing Surveys.

Volume 25, Issue 2, June 1993. pp. 73-169.

[31] C. Gupta, A. Mehta and U. Dayal. “PQR: Predicting Query Execution Times for

Autonomous Workload Management”. In Proc. of the 5th Intl. Conf. on Autonomic

Computing (ICAC’08). Chicago, IL, USA. June 2-6, 2008. pp. 13-22.

[32] C. Gupta, A. Mehta, S. Wang, and U. Dayal. “Fair, Effective, Efficient and Differentiated

Scheduling in an Enterprise Data Warehouse”. In Proc. of the 12th Intl. Conf. on Extending

Database Technology: Advances in Database Technology (EDBT’09). Saint Petersburg,

Russia. pp. 696-707.

[33] Greenplum. “Greenplum Database 4.2. Database Administrator Guide”.

http://media.gpadmin.me/wp-content/uploads/2012/11/GPDBAGuide.pdf

[34] J. Han and M. Kamber. “Data Mining: Concepts and Techniques” Morgan Kaufmann

Publishers, San Francisco, CA, USA, 2001.

[35] H. Heiss and R. Wagner. “Adaptive Load Control in Transaction Processing Systems”. In

Proc. of the 17th Intl. Conf. on Very Large Data Bases (VLDB’91). pp. 47-54.

[36] J. L. Hellerstein, Y. Diao, S. Parekh and D. M. Tilbury, “Feedback Control of Computing

Systems”, IEEE Press, Wiley-Interscience, John Wiley & Sons, Inc, 2004.

[37] J. M. Hellerstein, M. Stonebraker and J. Hamilton. “Architecture of a Database System”.

Foundations and Trends in Database. Volume 1, Issue 2, 2007. pp. 141-259.

[38] M. C. Huebscher and J. A. McCann. “A Survey of Autonomic Computing - Degrees,

Models, and Applications”. ACM Computing Surveys, Volume 40, Issue 3, August 2008,

Article No. 7.

[39] HP Corp. “HP Neoview Workload Management Services Guide”. July 2010.

http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c02455528-7.pdf

[40] IBM Corp. “Autonomic Computing: IBM‟s Perspective on the State of Information

Technology”. White Paper. 2001.

106

[41] IBM Corp. “z/OS Workload Manager: How it Works and How to Use it”. White Paper.

April 2004.

[42] IBM Corp. “DB2 Query Patroller Guide: Installation, Administration, and Usage”. Version

8.2. 2004.

[43] IBM Corp. “An Architectural Blueprint for Autonomic Computing”. White Paper. June

2006 (Fourth Edition).

[44] IBM Corp., “IBM DB2 for Linux, UNIX, and Windows; Best Practices: Implementing

DB2 Workload Management in a Data Warehouse”. 2011.

[45] IBM Corp. “IBM DB2 Database for Linux, UNIX, and Windows Information Center”. On-

line Documents. https://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp

[46] J. O. Kephart and D. M. Chess. “The Vision of Autonomic Computing”. IEEE Computer,

Volume 36, Issue 1, January 2003. pp. 41-50.

[47] J. O. Kephart, “Research Challenges of Autonomic Computing”. In Proc. of the 27th Intl.

Conf. on Software Engineering. St. Louis, Missouri, USA. May 15-21, 2005. pp. 15-22.

[48] J. O. Kephart and R. Das, “Achieving Self-Management via Utility Functions,” IEEE

Internet Computing. Volume 11, Issue 1, January 2007. pp. 40-48.

[49] L. Kleinrock. “Queueing Systems: Volume 1, Theory”. John Wiley & Sons, 1975.

[50] S. Krompass, H. Kuno, U. Dayal and A. Kemper, “Dynamic Workload Management for

Very Large Data Warehouses - Juggling Feathers and Bowling Balls”, In Proc. of 33rd

Intl.

Conf. on Very Large Data Bases (VLDB’07). Austria. pp. 1105-1115.

[51] S. Krompass, A. Scholz, M. C. Albutiu, H. Kuno, J. Wiener, U. Dayal and A. Kemper.

“Quality of Service-Enabled Management of Database Workloads”. In Special Issue of

IEEE Data Engineering Bulletin on Testing and Tuning of Database Systems, IEEE

Computer Society, 2008.

[52] S. Krompass, H. Kuno, J. L. Wiener, K. Wilkison, U. Dayal and A. Kemper, “Managing

Long-Running Queries”. In Proc. of the 12th Intl. Conf. on Extending Database

Technology: Advances in Database Technology (EDBT’09). Saint Petersburg, Russia. pp.

132-143.

107

[53] H. Kuno, U. Dayal, J. L. Wiener, K. Wilkinson, A. Ganapathi and S. Krompass.

“Managing Dynamic Mixed Workloads for Operational Business Intelligence”. Lecture

Notes in Computer Science, Databases in Networked Information Systems (DNIS’10),

March 2010. pp. 11-26.

[54] D. Kossmann, “The State of the Art in Distributed Query Processing”. In ACM Computing

Surveys, Volume 32, Issue 4, December 2000. pp. 422-469.

[55] E. Lazowska, J. Zahorjan, G. S. Graham and K. C. Sevcik “Quantitative System

Performance: Computer System Analysis Using Queueing Network Models”. Prentice-Hall

Inc., Englewood Cliffs, New Jersey, 1984.

[56] J. Li, R. V. Nehme and J. Naughton. “GSLPI: A Cost-Based Query Progress Indicator”. In

Proc. of the 28th Intl. Conf. on Data Engineering (ICDE’12). Washington, DC, USA. 2012.

pp. 678-689.

[57] S. Lightstone, G. Lohman and D. C. Zilio, “Toward Autonomic Computing with DB2

Universal Databases”. ACM SIGMOD Record, Volume 31 (3), September 2002. pp. 55-61.

[58] J. Lobo, R. Bhatia and S. Naqvi, “A Policy Description Language”, in Proc. of 16th

National Conf. on Artificial Intelligence, AAAI/IAAI, Orlando, Florida, USA, 1999.

pp. 291-298.

[59] G. Luo, J. F. Naughton, C. J. Ellmann, and M. W. Watzke. “Toward a Progress Indicator

for Database Queries”. In Proc. of the 2004 ACM SIGMOD Intl. Conf. on Management of

Data. Paris, France, June 13-18, 2004. pp. 791-802.

[60] G. Luo, J. F. Naughton, C. J. Ellmann, and M. W. Watzke. “Increasing the Accuracy and

Coverage of SQL Progress Indicators”. In Proc. of the 21st Intl. Conf. on Data Engineering

(ICDE’05). Tokyo, Japan. April 5-8, 2005. pp. 853-864.

[61] G. Luo, J. F. Naughton, and P. S. Yu. “Multi-query SQL Progress Indicators”. In Proc. of

the 10th Intl. Conf. on Extending Database Technology: Advances in Database Technology

(EDBT’06). Munich, Germany. March 26-31, 2006. pp. 921-941.

[62] P. Martin, M. Zhang, W. Powley, H. Boughton, P. Bird and R. Horman. “The Use of

Economic Models to Capture Importance Policy for Autonomic Database Management

108

Systems”. In Proc of the 1st ACM/IEEE Workshop on Autonomic Computing in Economics

in Conjunction with the 8th Intl. Conf. on Autonomic Computing (ICAC’11). 2011.

Germany. pp. 3-10.

[63] A. Mehta, C. Gupta and U. Dayal. “BI Batch Manager: A System for Managing Batch

Workloads on Enterprise Data-Warehouses”. In Proc. of the 11th Intl. Conf. on Extending

Database Technology: Advances in Database Technology (EDBT’08). Nantes, France.

March 25-29, 2008. pp. 640-651.

[64] A. Mehta, C. Gupta, S. Wang and U. Dayal. “Automated Workload Management for

Enterprise Data Warehouses”. In Bulletin of the IEEE Computer Society Technical

Committee on Data Engineering, Vol.31, No.1, March 2008. pp. 11-19.

[65] D. A. Menascé and J. O. Kephart, “Guest Editors' Introduction: Autonomic Computing”. In

IEEE Internet Computing, Volume 11, Issue 1, January 2007. pp. 18-21.

[66] Y. Meng, P. Bird, P. Martin and W. Powley, “An approach to managing the execution of

large SQL queries”. In Proc. of the Center for Advanced Studies on Collaborative

Research (CASCON’07). Toronto, Canada. October 22-25, 2007. pp. 268-271.

[67] Microsoft Corp. “Managing SQL Server Workloads with Resource Governor”. On-line

Documents. http://msdn.microsoft.com/en-us/library/bb933866.aspx

[68] Microsoft Corp., “Query Governor Cost Limit Option”. On-line Documents.

http://msdn.microsoft.com/en-us/library/ms190419.aspx

[69] A. Moenkeberg and G. Weikum, “Performance Evaluation of an Adaptive and Robust

Load Control Method for the Avoidance of Data Contention Thrashing”. In Proc of the 18th

Intl. Conf. on Very Large Data Bases (VLDB’92), Vancouver, BC, Canada, pp. 432- 443.

[70] J. Moffett and M. Sloman. “Policy Hierarchies for Distributed Systems Management”. In

IEEE Journal on Selected Areas in Communications, Volume 11, Issue 9, December, 1993.

[71] D. S. Moore and G. P. McCabe. “Introduction to the Practice of Statistics” (3rd

Edition). W.

H. Freeman & Co. and Sumanas, Inc., 1998.

[72] MySQL. http://www.mysql.com/

109

[73] D. Narayanan, E. Thereska and A. Ailamaki. “Continuous Resource Monitoring for Self-

Predicting DBMS”. In Proc. of the 13th IEEE Intl. Symposium on Modeling, Analysis, and

Simulation of Computer and Telecommunication Systems. Atlanta, Georgia, USA.

September 27-29, 2005. pp. 239-248.

[74] B. Niu, P. Martin, W. Powley, R. Horman, and P. Bird. “Workload Adaptation in

Autonomic DBMSs”, In Proc. of the Conf. of the Center for Advanced Studies on

Collaborative Research (CASCON’06). Toronto, Canada.

[75] B. Niu, P. Martin, W. Powley, P. Bird, and R. Horman. “Adapting Mixed Workloads to

Meet SLOs in Autonomic DBMSs”. In Proc. of the 2nd

Intl. Workshop on Self-managing

Database Systems in Conjunction with the 23rd

Intl. Conf. on Data Engineering (ICDE’07),

Istanbul, Turkey. April 15-20, 2007.

[76] B. Niu, P. Martin and W. Powley, “Towards Autonomic Workload Management in

DBMSs”. In Journal of Database Management, Volume 20, Issue 3, July-September 2009.

pp. 1-17.

[77] Oracle Corp. “Oracle Database Resource Manager”. On-line Documents.

http://download.oracle.com/docs/cd/B28359_01/server.111/b28310/dbrm.htm#i1010776

[78] H. H. Pang, M. J. Carey and M. Livny. “Managing Memory for Real-Time Queries”. In

Proc. of the ACM SIGMOD Intl. Conf. on Management of Data (SIGMOD’94).

Minneapolis, Minnesota, USA. May 24-27, 1994. pp. 221-232.

[79] S. Parekh, K. Rose, J. Hellerstein, S. Lightstone, M. Huras and V. Chang. “Managing the

Performance Impact of Administrative Utilities”. In Proc. of Self-Managing Distributed

Systems, Springer, Berlin, Heidelberg, February 19, 2004. pp. 130-142.

[80] S. Parekh, K. Rose, Y. Diao, V. Chang, J. Hellerstein, S. Lightstone, and M. Huras

“Throttling Utilities in the IBM DB2 Universal Database Server”, in Proc. of the 2004

American Control Conf., Boston, MA, USA. June 30-July 2, 2004.

[81] W. Powley, P. Martin and P. Bird, “DBMS Workload Control Using Throttling:

Experimental Insights”. In Proc. of the Conf. of the Center for Advanced Studies on

Collaborative Research (CASCON’08). Toronto, Canada. October 27-30, 2008. pp. 1-13.

110

[82] W. Powley, P. Martin, M. Zhang, P. Bird and K. McDonald. “Autonomic Workload

Execution Control Using Throttling”. In Proc. of the 5th Intl. Workshop on Self-managing

Database Systems in Conjunction with the 26th Intl. Conf. on Data Engineering (ICDE’10).

Long Beach, CA, USA. March 1-6, 2010.

[83] PostgreSQL. http://www.mysql.com/

[84] R. Ramakrishnan and J. Gehrke. “Database Management Systems” (3rd

Edition). McGraw-

Hill. 2003.

[85] S. M. Ross. “Introduction to Probability Models” (9th Edition). Elsevier Inc., 2007.

[86] B. Schroeder, M. Harchol-Balter, A. Iyengar, E. Nahum and A. Wierman. “How to

Determine a Good Multi-Programming Level for External Scheduling”. In Proc. of the 22nd

Intl. Conf. on Data Engineering (ICDE’06). April 3-7, 2006.

[87] B. Schroeder, M. Harchol-Balter, A. Iyengar, and E. Nahum and. “Achieving Class-Based

QoS for Transactional Workloads”. In Proc. of the 22nd

Intl. Conf. on Data Engineering

(ICDE’06). April 3-7, 2006.

[88] B. Schroeder, A. Wierman and M. Harchol-Balter. “Open vs. Closed: A Cautionary Tale”.

In Proc. of the 3rd

Conf. on Networked Systems Design & Implementation (NSDI'06). 2006.

[89] A. Silbershatz, H. Korth and S. Sudarshan. “Database System Concepts” (6th Edition).

McGraw-Hill. 2010.

[90] Teradata Corp. “Teradata Workload Analyzer User Guide”. On-line Documents.

http://www.info.teradata.com/eDownload.cfm?itemid=082330035

[91] Teradata Corp., “Teradata Dynamic Workload Manager”. On-line Documents.

http://info.teradata.com/edownload.cfm?itemid=082330034

[92] Transaction Processing Performance Council. http://www.tpc.org.

[93] W. E. Walsh, G. Tesauro, J. O. Kephart, R. Das. “Utility Functions in Autonomic

Systems”, In Proc. of the 1st Intl. Conf. on Autonomic Computing (ICAC’04). New York,

USA. May 17-18, 2004. pp. 70-77.

111

[94] T. Wasserman, P. Martin and H. Rizvi, “Sizing DB2 UDB Servers for Business

Intelligence Workloads”. In Proc. of the Conf. of the Center for Advanced Studies on

Collaborative Research (CASCON’04). Toronto, Canada. October, 2004. pp. 135-149.

[95] T. Wasserman, P. Martin, D. Skillicorn and H. Rizvi, “Using a Characterization of

Business Intelligence Workloads for Sizing New Database Systems”, Proc of ACM 7th Intl.

Workshop on Data Warehousing and OLAP, Washington, DC. Nov. 2004. pp. 7-13.

[96] G. Weikum, A. Moenkeberg, C. Hasse and P. Zabback. “Self-tuning Database Technology

and Information Services: from Wishful Thinking to Viable Engineering”. In Proc. of 28th

Intl. Conf. on Very Large Data Bases (VLDB’02), Hong Kong. August 2002. pp. 20-31.

[97] Weka - Data Mining Software in Java. http://www.cs.waikato.ac.nz/ml/weka/

[98] I. H. Witten and E. Frank. “Data Mining: Practical Machine Learning Tools and

Techniques” (2nd

Edition). Morgan Kaufmann Publishers, San Francisco, CA, USA, 2005.

[99] M. Zhang, P. Martin, W. Powley and P. Bird. “Using Economic Models to Allocate

Resources in Database Management Systems”. In Proc. of the Conf. of the Center for

Advanced Studies on Collaborative Research (CASCON’08), October 27-30, 2008,

Toronto, Canada. pp. 248-259.

[100] M. Zhang, B. Niu, P. Martin, W. Powley, P. Bird and K. McDonald. “Utility Function-

based Workload Management for DBMSs”. In Proc. of the 7th Intl. Conf. on Autonomic

and Autonomous Systems (ICAS’11). May 22-27, 2011. Venice, Italy. pp. 116-121.

[101] M. Zhang, B. Niu, P. Martin, W. Powley and P. Bird. “Utility Functions in Autonomic

Workload Management for DBMSs”. In International Journal on Advances in Intelligent

Systems. Vol. 5, No. 1&2, July 2012.

[102] M. Zhang, P. Martin, W. Powley, P. Bird, and K. McDonald. “Discovering Indicators for

Congestion in DBMSs”. In Proc. of the 7th Intl. Workshop on Self-managing Database

Systems in Conjunction with the 28th Intl. Conf. on Data Engineering (ICDE’12).

Washington, DC, USA. 2012. pp. 263-268.

112

[103] M. Zhang, P. Martin, W. Powley, P. Bird and D. Kalmuk. “A Framework for Autonomic

Workload Management in DBMSs”. A Special Issue on Autonomous Data Bases,

Information Technology, Oldenbourg Publishing Company, February 2014.

113

Appendix A

DB2 (v9.7) Monitoring Table Functions

Table 6.1 Information returned for MON_GET_WORKLOAD [45]

Column Name Data Type Description or corresponding

monitor element

WORKLOAD_NAME VARCHAR(128) workload_name - Workload

name

WORKLOAD_ID INTEGER workload_id - Workload ID

MEMBER SMALLINT member- Database member

ACT_ABORTED_TOTAL BIGINT act_aborted_total - Total aborted

activities

ACT_COMPLETED_TOTAL BIGINT act_completed_total - Total

completed activities

ACT_REJECTED_TOTAL BIGINT act_rejected_total - Total rejected

activities

AGENT_WAIT_TIME BIGINT agent_wait_time - Agent wait

time

AGENT_WAITS_TOTAL BIGINT agent_waits_total - Total agent

waits

POOL_DATA_L_READS BIGINT pool_data_l_reads - Buffer pool

data logical reads

POOL_INDEX_L_READS BIGINT pool_index_l_reads - Buffer pool

index logical reads

POOL_TEMP_DATA_L_READS BIGINT pool_temp_data_l_reads - Buffer

pool temporary data logical reads

POOL_TEMP_INDEX_L_READS BIGINT

pool_temp_index_l_reads -

Buffer pool temporary index

logical reads

POOL_TEMP_XDA_L_READS BIGINT

pool_temp_xda_l_reads - Buffer

pool temporary XDA data logical

reads

POOL_XDA_L_READS BIGINT pool_xda_l_reads - Buffer pool

XDA data logical reads

POOL_DATA_P_READS BIGINT pool_data_p_reads - Buffer pool

data physical reads

POOL_INDEX_P_READS BIGINT pool_index_p_reads - Buffer pool

index physical reads

POOL_TEMP_DATA_P_READS BIGINT

pool_temp_data_p_reads - Buffer

pool temporary data physical

reads

POOL_TEMP_INDEX_P_READS BIGINT pool_temp_index_p_reads -

114


physical reads

POOL_TEMP_XDA_P_READS BIGINT

pool_temp_xda_p_reads - Buffer

pool temporary XDA data

physical reads

POOL_XDA_P_READS BIGINT pool_xda_p_reads - Buffer pool

XDA data physical reads

POOL_DATA_WRITES BIGINT pool_data_writes - Buffer pool

data writes

POOL_INDEX_WRITES BIGINT pool_index_writes - Buffer pool

index writes

POOL_XDA_WRITES BIGINT pool_xda_writes - Buffer pool

XDA data writes

POOL_READ_TIME BIGINT pool_read_time - Total buffer

pool physical read time

POOL_WRITE_TIME BIGINT pool_write_time - Total buffer

pool physical write time

CLIENT_IDLE_WAIT_TIME BIGINT client_idle_wait_time - Client

idle wait time

DEADLOCKS BIGINT deadlocks - Deadlocks detected

DIRECT_READS BIGINT direct_reads - Direct reads from

database

DIRECT_READ_TIME BIGINT direct_read_time - Direct read

time

DIRECT_WRITES BIGINT direct_writes - Direct writes to

database

DIRECT_WRITE_TIME BIGINT direct_write_time - Direct write

time

DIRECT_READ_REQS BIGINT direct_read_reqs - Direct read

requests

DIRECT_WRITE_REQS BIGINT direct_write_reqs - Direct write

requests

FCM_RECV_VOLUME BIGINT fcm_recv_volume - FCM recv

volume

FCM_RECVS_TOTAL BIGINT fcm_recvs_total - FCM recvs

total

FCM_SEND_VOLUME BIGINT fcm_send_volume - FCM send

volume

FCM_SENDS_TOTAL BIGINT fcm_sends_total - FCM sends

total

FCM_RECV_WAIT_TIME BIGINT fcm_recv_wait_time - FCM recv

wait time

FCM_SEND_WAIT_TIME BIGINT fcm_send_wait_time - FCM send

wait time

IPC_RECV_VOLUME BIGINT ipc_recv_volume - Interprocess

115

communication recv volume

IPC_RECV_WAIT_TIME BIGINT ipc_recv_wait_time - Interprocess

communication recv wait time

IPC_RECVS_TOTAL BIGINT ipc_recvs_total - Interprocess

communication recvs total

IPC_SEND_VOLUME BIGINT ipc_send_volume - Interprocess

communication send volume

IPC_SEND_WAIT_TIME BIGINT

ipc_send_wait_time -

Interprocess communication send

wait time

IPC_SENDS_TOTAL BIGINT ipc_sends_total - Interprocess

communication send total

LOCK_ESCALS BIGINT lock_escals - Number of lock

escalations

LOCK_TIMEOUTS BIGINT lock_timeouts - Number of lock

timeouts

LOCK_WAIT_TIME BIGINT lock_wait_time - Time waited on

locks

LOCK_WAITS BIGINT lock_waits - Lock waits

LOG_BUFFER_WAIT_TIME BIGINT log_buffer_wait_time - Log

buffer wait time

NUM_LOG_BUFFER_FULL BIGINT num_log_buffer_full - Number of

full log buffers

LOG_DISK_WAIT_TIME BIGINT log_disk_wait_time - Log disk

wait time

LOG_DISK_WAITS_TOTAL BIGINT log_disk_waits_total - Log disk

waits total

RQSTS_COMPLETED_TOTAL BIGINT rqsts_completed_total - Total

requests completed

ROWS_MODIFIED BIGINT rows_modified - Rows modified

ROWS_READ BIGINT rows_read - Rows read

ROWS_RETURNED BIGINT rows_returned - Rows returned

TCPIP_RECV_VOLUME BIGINT tcpip_recv_volume - TCP/IP

received volume

TCPIP_SEND_VOLUME BIGINT tcpip_send_volume - TCP/IP

send volume

TCPIP_RECV_WAIT_TIME BIGINT tcpip_recv_wait_time - TCP/IP

recv wait time

TCPIP_RECVS_TOTAL BIGINT tcpip_recvs_total - TCP/IP recvs

total

TCPIP_SEND_WAIT_TIME BIGINT tcpip_send_wait_time - TCP/IP

send wait time

TCPIP_SENDS_TOTAL BIGINT tcpip_sends_total - TCP/IP sends

total

116

TOTAL_APP_RQST_TIME BIGINT total_app_rqst_time - Total

application request time

TOTAL_RQST_TIME BIGINT total_rqst_time - Total request

time

WLM_QUEUE_TIME_TOTAL BIGINT

wlm_queue_time_total -

Workload manager total queue

time

WLM_QUEUE_ASSIGNMENTS

_TOTAL BIGINT

wlm_queue_assignments_total -


assignments

TOTAL_CPU_TIME BIGINT total_cpu_time - Total CPU time

TOTAL_WAIT_TIME BIGINT total_wait_time - Total wait time

APP_RQSTS_COMPLETED_TOTAL BIGINT

app_rqsts_completed_total -

Total application requests

completed

TOTAL_SECTION_SORT_TIME BIGINT total_section_sort_time - Total

section sort time

TOTAL_SECTION_SORT_PROC_TIME BIGINT total_section_sort_proc_time -

Total section sort processing time

TOTAL_SECTION_SORTS BIGINT total_section_sorts - Total section

sorts

TOTAL_SORTS BIGINT total_sorts - Total Sorts

POST_THRESHOLD_SORTS BIGINT post_threshold_sorts - Post

threshold sorts

POST_SHRTHRESHOLD_SORTS BIGINT post_shrthreshold_sorts - Post

shared threshold sorts

SORT_OVERFLOWS BIGINT sort_overflows - Sort overflows

TOTAL_COMPILE_TIME BIGINT total_compile_time - Total

compile time

TOTAL_COMPILE_PROC_TIME BIGINT total_compile_proc_time - Total

compile processing time

TOTAL_COMPILATIONS BIGINT total_compilations - Total

compilations

TOTAL_IMPLICIT_COMPILE_TIME BIGINT total_implicit_compile_time -

Total implicit compile time

TOTAL_IMPLICIT_COMPILE_PROC_TIME BIGINT

total_implicit_compile_proc_time

- Total implicit compile

processing time

TOTAL_IMPLICIT_COMPILATIONS BIGINT total_implicit_compilations -

Total implicit complications

TOTAL_SECTION_TIME BIGINT total_section_time - Total section

time

TOTAL_SECTION_PROC_TIME BIGINT total_section_proc_time - Total

section processing time

TOTAL_APP_SECTION_EXECUTIONS BIGINT total_app_section_executions -

117

Total section executions

TOTAL_ACT_TIME BIGINT total_activity_time - Total

activity time

TOTAL_ACT_WAIT_TIME BIGINT total_activity_wait_time - Total

activity wait time

ACT_RQSTS_TOTAL BIGINT act_rqsts_total - Total activity

requests

TOTAL_ROUTINE_TIME BIGINT total_routine_time - Total routine

time

TOTAL_ROUTINE_INVOCATIONS BIGINT total_routine_invocations - Total

routine invocations

TOTAL_COMMIT_TIME BIGINT total_commit_time - Total

commit time

TOTAL_COMMIT_PROC_TIME BIGINT total_commit_proc_time - Total

commits processing time

TOTAL_APP_COMMITS BIGINT total_app_commits - Total

application commits

INT_COMMITS BIGINT int_commits - Internal commits

TOTAL_ROLLBACK_TIME BIGINT total_rollback_time - Total

rollback time

TOTAL_ROLLBACK_PROC_TIME BIGINT total_rollback_proc_time - Total

rollback processing time

TOTAL_APP_ROLLBACKS BIGINT total_app_rollbacks - Total

application rollbacks

INT_ROLLBACKS BIGINT int_rollbacks - Internal rollbacks

TOTAL_RUNSTATS_TIME BIGINT total_runstats_time - Total

runtime statistics

TOTAL_RUNSTATS_PROC_TIME BIGINT total_runstats_proc_time - Total

runtime statistics processing time

TOTAL_RUNSTATS BIGINT total_runstats - Total runtime

statistics

TOTAL_REORG_TIME BIGINT total_reorg_time - Total

reorganization time

TOTAL_REORG_PROC_TIME BIGINT total_reorg_proc_time - Total

reorganization processing time

TOTAL_REORGS BIGINT total_reorgs - Total

reorganizations

TOTAL_LOAD_TIME BIGINT total_load_time - Total load time

TOTAL_LOAD_PROC_TIME BIGINT total_load_proc_time - Total load

processing time

TOTAL_LOADS BIGINT total_loads - Total loads

CAT_CACHE_INSERTS BIGINT cat_cache_inserts - Catalog cache

inserts

CAT_CACHE_LOOKUPS BIGINT cat_cache_lookups - Catalog

118

cache lookups

PKG_CACHE_INSERTS BIGINT pkg_cache_inserts - Package

cache inserts

PKG_CACHE_LOOKUPS BIGINT pkg_cache_lookups - Package

cache lookups

THRESH_VIOLATIONS BIGINT thresh_violations - Number of

threshold violations

NUM_LW_THRESH_EXCEEDED BIGINT num_lw_thresh_exceeded -

Number of thresholds exceeded

IDA_SEND_WAIT_TIME BIGINT ida_send_wait_time - Time spent

waiting to send data

IDA_SENDS_TOTAL BIGINT ida_sends_total - Number of

times data sent

IDA_SEND_VOLUME BIGINT ida_send_volume - Total data

volume sent

IDA_RECV_WAIT_TIME BIGINT ida_recv_wait_time - Time spent

waiting to receive data

IDA_RECVS_TOTAL BIGINT ida_recvs_total - Number of

times data received

IDA_RECV_VOLUME BIGINT ida_recv_volume - Total data

volume received

ADDITIONAL_DETAILS BLOB(100K) Reserved for future use.

119

Table 6.2 Information returned for MON_GET_BUFFERPOOL [45]


monitor element

BP_NAME VARCHAR(128) bp_name - Buffer pool name


AUTOMATIC SMALLINT automatic - Buffer pool

automatic


database


requests


database


requests


data logical reads

POOL_TEMP_DATA_L_READS BIGINT

pool_temp_data_l_reads -

Buffer pool temporary data

logical reads

POOL_XDA_L_READS BIGINT pool_xda_l_reads - Buffer pool

XDA data logical reads




logical reads

POOL_INDEX_L_READS BIGINT pool_index_l_reads - Buffer

pool index logical reads




logical reads


data physical reads


pool_temp_data_p_reads -

Buffer pool temporary data

physical reads




pool_temp_xda_p_reads -

Buffer pool temporary XDA

data physical reads

POOL_INDEX_P_READS BIGINT pool_index_p_reads - Buffer

pool index physical reads

POOL_TEMP_INDEX_P_READS BIGINT

pool_temp_index_p_reads -


physical reads

120


data writes


XDA data writes


index writes


time


time





POOL_ASYNC_DATA_READS BIGINT pool_async_data_reads - Buffer

pool asynchronous data reads

POOL_ASYNC_DATA_READ_REQS BIGINT

pool_async_data_read_reqs -

Buffer pool asynchronous read

requests

POOL_ASYNC_DATA_WRITES BIGINT

pool_async_data_writes -

Buffer pool asynchronous data

writes

POOL_ASYNC_INDEX_READS BIGINT

pool_async_index_reads -

Buffer pool asynchronous index

reads

POOL_ASYNC_INDEX_READ_REQS BIGINT

pool_async_index_read_reqs -


read requests

POOL_ASYNC_INDEX_WRITES BIGINT

pool_async_index_writes -


writes

POOL_ASYNC_XDA_READS BIGINT

pool_async_xda_reads - Buffer

pool asynchronous XDA data

reads

POOL_ASYNC_XDA_READ_REQS BIGINT

pool_async_xda_read_reqs -

Buffer pool asynchronous XDA

read requests

POOL_ASYNC_XDA_WRITES BIGINT

pool_async_xda_writes - Buffer

pool asynchronous XDA data

writes

POOL_NO_VICTIM_BUFFER BIGINT pool_no_victim_buffer - Buffer

pool no victim buffers

POOL_LSN_GAP_CLNS BIGINT pool_lsn_gap_clns - Buffer pool

log space cleaners triggered

POOL_DRTY_PG_STEAL_CLNS BIGINT

pool_drty_pg_steal_clns -

Buffer pool victim page

cleaners triggered

121

POOL_DRTY_PG_THRSH_CLNS BIGINT

pool_drty_pg_thrsh_clns -

Buffer pool threshold cleaners

triggered

VECTORED_IOS BIGINT vectored_ios - Number of

vectored IO requests

PAGES_FROM_VECTORED_IOS BIGINT

pages_from_vectored_ios -

Total number of pages read by

vectored IO

BLOCK_IOS BIGINT block_ios - Number of block IO

requests

PAGES_FROM_BLOCK_IOS BIGINT

pages_from_block_ios - Total

number of pages read by block

IO

UNREAD_PREFETCH_PAGES BIGINT unread_prefetch_pages - Unread

prefetch pages

FILES_CLOSED BIGINT files_closed - Database files

closed

POOL_DATA_GBP_L_READS BIGINT Reserved for future use

POOL_DATA_GBP_P_READS BIGINT Reserved for future use

POOL_DATA_LBP_PAGES_FOUND BIGINT Reserved for future use

POOL_DATA_GBP_INVALID_PAGES BIGINT Reserved for future use

POOL_INDEX_GBP_L_READS BIGINT Reserved for future use

POOL_INDEX_GBP_P_READS BIGINT Reserved for future use

POOL_INDEX_LBP_PAGES_FOUND BIGINT Reserved for future use

POOL_INDEX_GBP_INVALID_PAGES BIGINT Reserved for future use

POOL_ASYNC_DATA_GBP_L_READS BIGINT Reserved for future use

POOL_ASYNC_DATA_GBP_P_READS BIGINT Reserved for future use

POOL_ASYNC_DATA_LBP_PAGES_FOUN

D BIGINT Reserved for future use

POOL_ASYNC_DATA_GBP_INVALID_PA

GES BIGINT Reserved for future use

POOL_ASYNC_INDEX_GBP_L_READS BIGINT Reserved for future use

POOL_ASYNC_INDEX_GBP_P_READS BIGINT Reserved for future use

POOL_ASYNC_INDEX_LBP_PAGES_FOU

ND BIGINT Reserved for future use

POOL_ASYNC_INDEX_GBP_INVALID_PA

GES BIGINT Reserved for future use

POOL_XDA_GBP_L_READS BIGINT Reserved for future use

POOL_XDA_GBP_P_READS BIGINT Reserved for future use

POOL_XDA_LBP_PAGES_FOUND BIGINT Reserved for future use

POOL_XDA_GBP_INVALID_PAGES BIGINT Reserved for future use

POOL_ASYNC_XDA_GBP_L_READS BIGINT Reserved for future use

122

POOL_ASYNC_XDA_GBP_P_READS BIGINT Reserved for future use

POOL_ASYNC_XDA_LBP_PAGES_FOUND BIGINT Reserved for future use

POOL_ASYNC_XDA_GBP_INVALID_PAG

ES BIGINT Reserved for future use

POOL_ASYNC_READ_TIME BIGINT

pool_async_read_time - Buffer

Pool Asynchronous Read Time

monitor element

POOL_ASYNC_WRITE_TIME BIGINT

pool_async_write_time - Buffer

pool asynchronous write time

monitor element

BP_CUR_BUFFSZ BIGINT bp_cur_buffsz - Current Size of

Buffer Pool monitor element

ADDITIONAL_DETAILS BLOB(100K) Reserved for future use

123

Table 6.3 Information returned for MON_GET_TABLESPACE [45]


monitor element

TBSP_NAME VARCHAR(128) tablespace_name - Table space name

TBSP_ID BIGINT tablespace_id - Table space

identification


TBSP_TYPE VARCHAR(10)

tablespace_type - Table space type. This

interface returns a text identifier based

on defines in sqlutil.h, and is one of:

DMS

SMS

TBSP_CONTENT_TYPE VARCHAR(10)

tablespace_content_type - Table space

content type. This interface returns a

text identifier based on defines in

sqlmon.h, and is one of:

ANY

LARGE

SYSTEMP

USRTEMP

TBSP_PAGE_SIZE BIGINT tablespace_page_size - Table space

page size

TBSP_EXTENT_SIZE BIGINT tablespace_extent_size - Table space

extent size

TBSP_PREFETCH_SIZE BIGINT tablespace_prefetch_size - Table space

prefetch size

TBSP_CUR_POOL_ID BIGINT tablespace_cur_pool_id - Buffer pool

currently being used

TBSP_NEXT_POOL_ID BIGINT tablespace_next_pool_id - Buffer pool

that will be used at next startup

FS_CACHING SMALLINT fs_caching - File system caching

TBSP_REBALANCER_MODE VARCHAR(30)

tablespace_rebalancer_mode -

Rebalancer mode. This interface returns

a text identifier based on defines in

sqlmon.h, and is one of:

NO_REBAL

FWD_REBAL

REV_REBAL

FWD_REBAL_OF_2PASS

REV_REBAL_OF_2PASS

TBSP_USING_AUTO_STORAGE SMALLINT tablespace_using_auto_storage - Table

space enabled for automatic storage

TBSP_AUTO_RESIZE_ENABLED SMALLINT tablespace_auto_resize_enabled - Table

space automatic resizing enabled


database

124

DIRECT_READ_REQS BIGINT direct_read_reqs - Direct read requests

DIRECT_WRITES BIGINT direct_writes - Direct writes to database

DIRECT_WRITE_REQS BIGINT direct_write_reqs - Direct write requests

POOL_DATA_L_READS BIGINT pool_data_l_reads - Buffer pool data

logical reads

POOL_TEMP_DATA_L_READS BIGINT pool_temp_data_l_reads - Buffer pool

temporary data logical reads

POOL_XDA_L_READS BIGINT pool_xda_l_reads - Buffer pool XDA

data logical reads

POOL_TEMP_XDA_L_READS BIGINT pool_temp_xda_l_reads - Buffer pool

temporary XDA data logical reads

POOL_INDEX_L_READS BIGINT pool_index_l_reads - Buffer pool index

logical reads

POOL_TEMP_INDEX_L_READS BIGINT pool_temp_index_l_reads - Buffer pool

temporary index logical reads

POOL_DATA_P_READS BIGINT pool_data_p_reads - Buffer pool data

physical reads

POOL_TEMP_DATA_P_READS BIGINT pool_temp_data_p_reads - Buffer pool

temporary data physical reads

POOL_XDA_P_READS BIGINT pool_xda_p_reads - Buffer pool XDA

data physical reads

POOL_TEMP_XDA_P_READS BIGINT pool_temp_xda_p_reads - Buffer pool

temporary XDA data physical reads

POOL_INDEX_P_READS BIGINT pool_index_p_reads - Buffer pool index

physical reads

POOL_TEMP_INDEX_P_READS BIGINT pool_temp_index_p_reads - Buffer pool

temporary index physical reads

POOL_DATA_WRITES BIGINT pool_data_writes - Buffer pool data

writes

POOL_XDA_WRITES BIGINT pool_xda_writes - Buffer pool XDA

data writes

POOL_INDEX_WRITES BIGINT pool_index_writes - Buffer pool index

writes

DIRECT_READ_TIME BIGINT direct_read_time - Direct read time

DIRECT_WRITE_TIME BIGINT direct_write_time - Direct write time

POOL_READ_TIME BIGINT pool_read_time - Total buffer pool

physical read time

POOL_WRITE_TIME BIGINT pool_write_time - Total buffer pool

physical write time

POOL_ASYNC_DATA_READS BIGINT pool_async_data_reads - Buffer pool

asynchronous data reads

POOL_ASYNC_DATA_READ_REQS BIGINT pool_async_data_read_reqs - Buffer

pool asynchronous read requests

POOL_ASYNC_DATA_WRITES BIGINT pool_async_data_writes - Buffer pool

125

asynchronous data writes

POOL_ASYNC_INDEX_READS BIGINT pool_async_index_reads - Buffer pool

asynchronous index reads

POOL_ASYNC_INDEX_READ_REQS BIGINT pool_async_index_read_reqs - Buffer

pool asynchronous index read requests

POOL_ASYNC_INDEX_WRITES BIGINT pool_async_index_writes - Buffer pool

asynchronous index writes

POOL_ASYNC_XDA_READS BIGINT pool_async_xda_reads - Buffer pool

asynchronous XDA data reads

POOL_ASYNC_XDA_READ_REQS BIGINT pool_async_xda_read_reqs - Buffer

pool asynchronous XDA read requests

POOL_ASYNC_XDA_WRITES BIGINT pool_async_xda_writes - Buffer pool

asynchronous XDA data writes

VECTORED_IOS BIGINT vectored_ios - Number of vectored IO

requests

PAGES_FROM_VECTORED_IOS BIGINT pages_from_vectored_ios - Total

number of pages read by vectored IO


requests

PAGES_FROM_BLOCK_IOS BIGINT pages_from_block_ios - Total number

of pages read by block IO

UNREAD_PREFETCH_PAGES BIGINT unread_prefetch_pages - Unread

prefetch pages

FILES_CLOSED BIGINT files_closed - Database files closed

TBSP_STATE VARCHAR(256) tablespace_state - Table space state

TBSP_USED_PAGES BIGINT tablespace_used_pages - Used pages in

table space

TBSP_FREE_PAGES BIGINT tablespace_free_pages - Free pages in

table space

TBSP_USABLE_PAGES BIGINT tablespace_usable_pages - Usable pages

in table space

TBSP_TOTAL_PAGES BIGINT tablespace_total_pages - Total pages in

table space

TBSP_PENDING_FREE_PAGES BIGINT tablespace_pending_free_pages -

Pending free pages in table space

TBSP_PAGE_TOP BIGINT tablespace_page_top - Table space high

watermark

TBSP_MAX_PAGE_TOP BIGINT tbsp_max_page_top - Maximum table

space page high watermark

RECLAIMABLE_SPACE_ENABLED SMALLINT reclaimable_space_enabled -

Reclaimable space enabled indicator

AUTO_STORAGE_HYBRID SMALLINT auto_storage_hybrid - Hybrid automatic

storage table space indicator

TBSP_PATHS_DROPPED SMALLINT tablespace_paths_dropped - Table space

126

using dropped path

POOL_DATA_GBP_L_READS BIGINT Reserved for future use.

POOL_DATA_GBP_P_READS BIGINT Reserved for future use.

POOL_DATA_LBP_PAGES_FOUND BIGINT Reserved for future use.

POOL_DATA_GBP_INVALID_PAGES BIGINT Reserved for future use.

POOL_INDEX_GBP_L_READS BIGINT Reserved for future use.

POOL_INDEX_GBP_P_READS BIGINT Reserved for future use.

POOL_INDEX_LBP_PAGES_FOUND BIGINT Reserved for future use.

POOL_INDEX_GBP_INVALID_PAGES BIGINT Reserved for future use.

POOL_ASYNC_DATA_GBP_L_READS BIGINT Reserved for future use.

POOL_ASYNC_DATA_GBP_P_READS BIGINT Reserved for future use.

POOL_ASYNC_DATA_LBP_PAGES_FOUN

D BIGINT Reserved for future use.

POOL_ASYNC_DATA_GBP_INVALID_PA

GES BIGINT Reserved for future use.

POOL_ASYNC_INDEX_GBP_L_READS BIGINT Reserved for future use.

POOL_ASYNC_INDEX_GBP_P_READS BIGINT Reserved for future use.

POOL_ASYNC_INDEX_LBP_PAGES_FOU

ND BIGINT Reserved for future use.

POOL_ASYNC_INDEX_GBP_INVALID_PA

GES BIGINT Reserved for future use.

TABLESPACE_MIN_RECOVERY_TIME TIMESTAMP Reserved for future use.

DBPARTITIONNUM SMALLINT

In a partitioned database environment,

this is the numeric identifier for the

database member. For DB2® Enterprise

Server Edition and in a DB2

pureScale® environment, this value is

0.

POOL_XDA_GBP_L_READS BIGINT Reserved for future use.

POOL_XDA_GBP_P_READS BIGINT Reserved for future use.

POOL_XDA_LBP_PAGES_FOUND BIGINT Reserved for future use.

POOL_XDA_GBP_INVALID_PAGES BIGINT Reserved for future use.

POOL_ASYNC_XDA_GBP_L_READS BIGINT Reserved for future use.

POOL_ASYNC_XDA_GBP_P_READS BIGINT Reserved for future use.

POOL_ASYNC_XDA_LBP_PAGES_FOUND BIGINT Reserved for future use.

POOL_ASYNC_XDA_GBP_INVALID_PAG

ES BIGINT Reserved for future use.

POOL_ASYNC_READ_TIME BIGINT pool_async_read_time - Buffer pool

asynchronous read time .

POOL_ASYNC_WRITE_TIME BIGINT pool_async_write_time - Buffer pool

asynchronous write time .

127

TBSP_TRACKMOD_STATE VARCHAR(32)

tbsp_trackmod_state - Table space

trackmod state . This interface returns a

text identifier, and is one of:

CLEAN

DIRTY

ININCREMENTAL

READFULL

READINCREMENTAL

UNAVAILABLE


128

Table 6.4 Information Returned for MON_GET_CONTAINER [45]


monitor element

TBSP_NAME VARCHAR(128) tablespace_name - Table space name

TBSP_ID BIGINT tablespace_id - Table space

identification

CONTAINER_NAME VARCHAR(256) container_name - Container name

CONTAINER_ID BIGINT container_id - Container identification


CONTAINER_TYPE VARCHAR(16) container_type - Container type

STRIPE_SET BIGINT container_stripe_set - Stripe set


database

DIRECT_WRITES BIGINT direct_writes - Direct writes to database

DIRECT_READ_TIME BIGINT direct_read_time - Direct read time

DIRECT_WRITE_TIME BIGINT direct_write_time - Direct write time

PAGES_READ BIGINT pages_read - Number of pages read

PAGES_WRITTEN BIGINT pages_written - Number of pages

written

VECTORED_IOS BIGINT vectored_ios - Number of vectored IO

requests

PAGES_FROM_VECTORED_IOS BIGINT pages_from_vectored_ios - Total

number of pages read by vectored IO


requests

PAGES_FROM_BLOCK_IOS BIGINT pages_from_block_ios - Total number

of pages read by block IO

POOL_READ_TIME BIGINT pool_read_time - Total buffer pool

physical read time

POOL_WRITE_TIME BIGINT pool_write_time - Total buffer pool

physical write time

TOTAL_PAGES BIGINT container_total_pages - Total pages in

container

USABLE_PAGES BIGINT container_usable_pages - Usable pages

in container

ACCESSIBLE SMALLINT container_accessible - Accessibility of

container

FS_ID VARCHAR(22) fs_id - Unique file system identification

number

FS_TOTAL_SIZE BIGINT fs_total_size - Total size of a file

system

129

FS_USED_SIZE BIGINT fs_used_size - Amount of space used

on a file system


130

Table 6.5 Information returned for MON_GET_SERVICE_SUBCLASS [45]


monitor element

SERVICE_SUPERCLASS_NAME VARCHAR(128) service_superclass_name -

Service superclass name

SERVICE_SUBCLASS_NAME VARCHAR(128) service_subclass_name - Service

subclass name

SERVICE_CLASS_ID INTEGER service_class_id - Service class

ID


ACT_ABORTED_TOTAL BIGINT act_aborted_total - Total aborted

activities

ACT_COMPLETED_TOTAL BIGINT act_completed_total - Total

completed activities

ACT_REJECTED_TOTAL BIGINT act_rejected_total - Total rejected

activities

AGENT_WAIT_TIME BIGINT agent_wait_time - Agent wait

time

AGENT_WAITS_TOTAL BIGINT agent_waits_total - Total agent

waits


data logical reads

POOL_INDEX_L_READS BIGINT pool_index_l_reads - Buffer pool

index logical reads

POOL_TEMP_DATA_L_READS BIGINT pool_temp_data_l_reads - Buffer

pool temporary data logical reads




logical reads




reads

POOL_XDA_L_READS BIGINT



reads


data physical reads

POOL_INDEX_P_READS BIGINT pool_index_p_reads - Buffer pool

index physical reads


pool_temp_data_p_reads - Buffer

pool temporary data physical

reads

POOL_TEMP_INDEX_P_READS BIGINT

pool_temp_index_p_reads -


physical reads

131


pool_temp_xda_p_reads - Buffer


physical reads




data writes


index writes


XDA data writes





CLIENT_IDLE_WAIT_TIME BIGINT client_idle_wait_time - Client

idle wait time

DEADLOCKS BIGINT deadlocks - Deadlocks detected


database


time


database


time


requests


requests

FCM_RECV_VOLUME BIGINT fcm_recv_volume - FCM recv

volume

FCM_RECVS_TOTAL BIGINT fcm_recvs_total - FCM recvs

total

FCM_SEND_VOLUME BIGINT fcm_send_volume - FCM send

volume

FCM_SENDS_TOTAL BIGINT fcm_sends_total - FCM sends

total

FCM_RECV_WAIT_TIME BIGINT fcm_recv_wait_time - FCM recv

wait time

FCM_SEND_WAIT_TIME BIGINT fcm_send_wait_time - FCM send

wait time

IPC_RECV_VOLUME BIGINT ipc_recv_volume - Interprocess

communication recv volume

IPC_RECV_WAIT_TIME BIGINT ipc_recv_wait_time - Interprocess

132

communication recv wait time

IPC_RECVS_TOTAL BIGINT ipc_recvs_total - Interprocess

communication recvs total

IPC_SEND_VOLUME BIGINT ipc_send_volume - Interprocess

communication send volume

IPC_SEND_WAIT_TIME BIGINT

ipc_send_wait_time -

Interprocess communication send

wait time

IPC_SENDS_TOTAL BIGINT ipc_sends_total - Interprocess

communication send total

LOCK_ESCALS BIGINT lock_escals - Number of lock

escalations

LOCK_TIMEOUTS BIGINT lock_timeouts - Number of lock

timeouts

LOCK_WAIT_TIME BIGINT lock_wait_time - Time waited on

locks

LOCK_WAITS BIGINT lock_waits - Lock waits

LOG_BUFFER_WAIT_TIME BIGINT log_buffer_wait_time - Log

buffer wait time

NUM_LOG_BUFFER_FULL BIGINT num_log_buffer_full - Number of

full log buffers

LOG_DISK_WAIT_TIME BIGINT log_disk_wait_time - Log disk

wait time

LOG_DISK_WAITS_TOTAL BIGINT log_disk_waits_total - Log disk

waits total

RQSTS_COMPLETED_TOTAL BIGINT rqsts_completed_total - Total

requests completed

ROWS_MODIFIED BIGINT rows_modified - Rows modified

ROWS_READ BIGINT rows_read - Rows read

ROWS_RETURNED BIGINT rows_returned - Rows returned

TCPIP_RECV_VOLUME BIGINT tcpip_recv_volume - TCP/IP

received volume

TCPIP_SEND_VOLUME BIGINT tcpip_send_volume - TCP/IP

send volume

TCPIP_RECV_WAIT_TIME BIGINT tcpip_recv_wait_time - TCP/IP

recv wait time

TCPIP_RECVS_TOTAL BIGINT tcpip_recvs_total - TCP/IP recvs

total

TCPIP_SEND_WAIT_TIME BIGINT tcpip_send_wait_time - TCP/IP

send wait time

TCPIP_SENDS_TOTAL BIGINT tcpip_sends_total - TCP/IP sends

total

TOTAL_APP_RQST_TIME BIGINT total_app_rqst_time - Total

application request time

133

TOTAL_RQST_TIME BIGINT total_rqst_time - Total request

time

WLM_QUEUE_TIME_TOTAL BIGINT

wlm_queue_time_total -


time

WLM_QUEUE_ASSIGNMENTS

_TOTAL BIGINT

wlm_queue_assignments_total -


assignments

TOTAL_RQST_MAPPED_IN BIGINT total_rqst_mapped_in - Total

request mapped-in

TOTAL_RQST_MAPPED_OUT BIGINT total_rqst_mapped_out - Total

request mapped-out

TOTAL_CPU_TIME BIGINT total_cpu_time - Total CPU time

TOTAL_WAIT_TIME BIGINT total_wait_time - Total wait time

APP_RQSTS_COMPLETED_TOTAL BIGINT

app_rqsts_completed_total -

Total application requests

completed

TOTAL_SECTION_SORT_TIME BIGINT total_section_sort_time - Total

section sort time

TOTAL_SECTION_SORT_PROC_TIME BIGINT total_section_sort_proc_time -

Total section sort processing time

TOTAL_SECTION_SORTS BIGINT total_section_sorts - Total section

sorts

TOTAL_SORTS BIGINT total_sorts - Total Sorts

POST_THRESHOLD_SORTS BIGINT post_threshold_sorts - Post

threshold sorts

POST_SHRTHRESHOLD_SORTS BIGINT post_shrthreshold_sorts - Post

shared threshold sorts

SORT_OVERFLOWS BIGINT sort_overflows - Sort overflows

TOTAL_COMPILE_TIME BIGINT total_compile_time - Total

compile time

TOTAL_COMPILE_PROC_TIME BIGINT total_compile_proc_time - Total

compile processing time

TOTAL_COMPILATIONS BIGINT total_compilations - Total

compilations

TOTAL_IMPLICIT_COMPILE_TIME BIGINT total_implicit_compile_time -

Total implicit compile time

TOTAL_IMPLICIT_COMPILE_PROC_TIME BIGINT

total_implicit_compile_proc_time

- Total implicit compile

processing time

TOTAL_IMPLICIT_COMPILATIONS BIGINT total_implicit_compilations -

Total implicit complications

TOTAL_SECTION_TIME BIGINT total_section_time - Total section

time

TOTAL_SECTION_PROC_TIME BIGINT total_section_proc_time - Total

134

section processing time

TOTAL_APP_SECTION_EXECUTIONS BIGINT total_app_section_executions -

Total section executions

TOTAL_ACT_TIME BIGINT total_act_time - Total activity

time

TOTAL_ACT_WAIT_TIME BIGINT total_act_wait_time - Total

activity wait time

ACT_RQSTS_TOTAL BIGINT act_rqsts_total - Total activity

requests

TOTAL_ROUTINE_TIME BIGINT total_routine_time - Total routine

time

TOTAL_ROUTINE_INVOCATIONS BIGINT total_routine_invocations - Total

routine invocations

TOTAL_COMMIT_TIME BIGINT total_commit_time - Total

commit time

TOTAL_COMMIT_PROC_TIME BIGINT total_commit_proc_time - Total

commits processing time

TOTAL_APP_COMMITS BIGINT total_app_commits - Total

application commits

INT_COMMITS BIGINT int_commits - Internal commits

TOTAL_ROLLBACK_TIME BIGINT total_rollback_time - Total

rollback time

TOTAL_ROLLBACK_PROC_TIME BIGINT total_rollback_proc_time - Total

rollback processing time

TOTAL_APP_ROLLBACKS BIGINT total_app_rollbacks - Total

application rollbacks

INT_ROLLBACKS BIGINT int_rollbacks - Internal rollbacks

TOTAL_RUNSTATS_TIME BIGINT total_runstats_time - Total

runtime statistics

TOTAL_RUNSTATS_PROC_TIME BIGINT total_runstats_proc_time - Total

runtime statistics processing time

TOTAL_RUNSTATS BIGINT total_runstats - Total runtime

statistics

TOTAL_REORG_TIME BIGINT total_reorg_time - Total

reorganization time

TOTAL_REORG_PROC_TIME BIGINT total_reorg_proc_time - Total

reorganization processing time

TOTAL_REORGS BIGINT total_reorgs - Total

reorganizations

TOTAL_LOAD_TIME BIGINT total_load_time - Total load time

TOTAL_LOAD_PROC_TIME BIGINT total_load_proc_time - Total load

processing time

TOTAL_LOADS BIGINT total_loads - Total loads

CAT_CACHE_INSERTS BIGINT cat_cache_inserts - Catalog cache

135

inserts

CAT_CACHE_LOOKUPS BIGINT cat_cache_lookups - Catalog

cache lookups

PKG_CACHE_INSERTS BIGINT pkg_cache_inserts - Package

cache inserts

PKG_CACHE_LOOKUPS BIGINT pkg_cache_lookups - Package

cache lookups

THRESH_VIOLATIONS BIGINT thresh_violations - Number of

threshold violations

NUM_LW_THRESH_EXCEEDED BIGINT num_lw_thresh_exceeded -

Number of thresholds exceeded

IDA_SEND_WAIT_TIME BIGINT ida_send_wait_time - Time spent

waiting to send data

IDA_SENDS_TOTAL BIGINT ida_sends_total - Number of

times data sent

IDA_SEND_VOLUME BIGINT ida_send_volume - Total data

volume sent

IDA_RECV_WAIT_TIME BIGINT ida_recv_wait_time - Time spent

waiting to receive data

IDA_RECVS_TOTAL BIGINT ida_recvs_total - Number of

times data received

IDA_RECV_VOLUME BIGINT ida_recv_volume - Total data

volume received

ADDITIONAL_DETAILS BLOB(100K) Reserved for future use

136

Appendix B

TPC-H Benchmark

The Transaction Processing Performance Council (TPC) is a non-profit corporation founded to

define transaction processing and database benchmarks and to disseminate objective, verifiable

TPC performance data to industry. One of the TPC database benchmarks is the TPC

BenchmarkTM

H (TPC-H), which is a decision support benchmark. It consists of a suite of

business oriented ad-hoc queries and concurrent data modifications. The queries and the data

populating the database have been chosen to have broad industry-wide relevance. This

benchmark illustrates decision support systems that examine large volumes of data, execute

queries with a high degree of complexity, and give answers to critical business questions [92].

Table 6.6 Tables and Cardinalities in TPC-H Benchmark [92]

Table Name Cardinality

(in rows)

Length of Typical

Row (in bytes)

Typical Table

Size (in MB)

SUPPLIER 10,000*SF 159 2

PART 200,000*SF 155 30

PARTSUPP 800,000*SF 144 110

CUSTOMER 150,000*SF 179 26

ORDERS 1,500,000*SF 104 149

LINEITEM 6,001,215*SF 112 641

NATION 25 128 < 1

REGION 5 124 < 1

137

The TPC-H database consists of eight separate and individual tables listed in Table 6.6.

Scale factors used for the test database are chosen from the set of fixed scale factors defined as

follows: 1, 10, 30, 100, 300, 1,000, 3,000, and 10,000. The minimum required scale factor for a

test database is SF=1, where the database is approximately 1GB. Table 6.7 lists TPC-H query‟s

number, name, and corresponding business question.

Table 6.7 Queries in TPC-H Benchmark [92]

Number Name Business Question

Q1 Pricing Summary

Report Query Reports the amount of business that was billed, shipped, and returned

Q2 Minimum Cost

Supplier Query

Finds which supplier should be selected to place an order for a given

part in a given region

Q3 Shipping Priority

Query This query retrieves the 10 unshipped orders with the highest value

Q4 Order Priority

Checking Query

Determines how well the order priority system is working and gives

an assessment of customer satisfaction

Q5 Local Supplier

Volume Query Lists the revenue volume done through local suppliers

Q6 Forecasting Revenue

Change Query

Quantifies the amount of revenue increase that would have resulted

from eliminating certain company-wide discount in a given

percentage range in a given year

Q7 Volume Shipping

Query

Determines the value of goods shipped between certain nations to

help in the re-negotiation of shipping contracts

Q8 National Market Share

Query

Determines how the market share of a given nation within a given

region has changed over two years for a given part type

Q9 Product Type Profit

Measure Query

Determines how much profit is made on a given line of parts, broken

out by supplier nation and year

Q10 Returned Item

Reporting Query

Identifies customers who might be having problems with the parts

that are shipped to them

138

Q11 Important Stock

Identification Query Finds the most important subset of suppliers' stock in a given nation

Q12 Shipping Modes and

Order Priority Query

Determines whether selecting less expensive modes of shipping is

negatively affecting the critical-priority orders by causing more parts

to be received by customers after the committed date

Q13 Customer Distribution

Query Seeks relationships between customers and the size of their orders

Q14 Promotion Effect

Query

Monitors the market response to a promotion such as TV

advertisements or a special campaign

Q15 Top Supplier Query Determines the top supplier so it can be rewarded, given more

business, or identified for special recognition

Q16 Parts/Supplier

Relationship Query Finds out how many suppliers can supply parts with given attributes

Q17 Small-Quantity-Order

Revenue Query

Determines how much average yearly revenue would be lost if

orders were no longer filled for small quantities of certain parts

Q18 Large Volume

Customer Query Ranks customers based on their having placed a large quantity order

Q19 Discounted Revenue

Query

Reports the gross discounted revenue attributed to the sale of

selected parts handled in a particular manner

Q20 Potential Part

Promotion Query

Identifies suppliers in a particular nation having selected parts that

may be candidates for a promotional offer

Q21 Suppliers Who Kept

Orders Waiting Query

Identifies certain suppliers who were not able to ship required parts

in a timely manner

Q22 Global Sales

Opportunity Query

Identifies geographies where there are customers who may be likely

to make a purchase

139

Appendix C

TPC-C Benchmark

TPC BenchmarkTM

C (TPC-C) is an on-line transaction processing (OLTP) benchmark. TPC-C

simulates a complete computing environment where a population of users executes transactions

against a database. The benchmark is centered on the principal activities (transactions) of an

order-entry environment. These transactions include entering and delivering orders, recording

payments, checking the status of orders, and monitoring the level of stock at the warehouses.

While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the

activity of any particular business segment, but, rather represents any industry that must manage,

sell, or distribute a product or service [92].

The components of the TPC-C database are defined to consist of nine separate and

individual tables. The relationships among these tables are defined in the entity-relationship

diagram shown in Figure 24.

Figure 6.1 Database Schema for TPC-C Benchmark [92]

Warehouse Dis trict

His tory

Customer

New-Order

OrderOrder-L ineItem

Stock

W W*10

3k

1+

W*30k

W*30k+5-15

0-1

1+W*30k+

W*9k+

W*300k+

3+

100k

W

W*100k

100k

10

140

All numbers shown In Figure 24 illustrate the database population requirements. The

numbers in the entity blocks represent the cardinality of the tables (number of rows). These

numbers are factored by W, the number of Warehouses, to illustrate the database scaling. The

numbers next to the relationship arrows represent the cardinality of the relationships (average

number of children per parent). The plus (+) symbol is used after the cardinality of a relationship

or table to illustrate that this number is subject to small variations in the initial database

population over the measurement interval as rows are added or deleted.

The mixed five different concurrent transactions of TPC-C [92]:

New-Order: consists of entering a complete order through a single database transaction. It

represents a mid-weight, read-write transaction with a high frequency of execution and

stringent response time requirements to satisfy on-line users. This transaction is the

backbone of the workload. It is designed to place a variable load on the system to reflect

on-line database activity as typically found in production environments.

Payment: updates the customer's balance and reflects the payment on the district and

warehouse sales statistics. It represents a light-weight, read-write transaction with a high

frequency of execution and stringent response time requirements to satisfy on-line users. In

addition, this transaction includes non-primary key access to the CUSTOMER table.

Order-Status: queries the status of a customer's last order. It represents a mid-weight read-

only database transaction with a low frequency of execution and response time requirement

to satisfy on-line users.

Delivery: consists of processing a batch of 10 new (not yet delivered) orders. Each

order is processed (delivered) in full within the scope of a read-write database

141

transaction. The number of orders delivered as a group (or batched) within the same

database transaction is implementation specific. The business transaction

comprised of one or more (up to 10) database transactions, has a low frequency of

execution and complete within a relaxed response time requirement. The Delivery

transaction is intended to be executed in deferred mode through a queuing

mechanism, rather than interactively, with terminal response indicating transaction

completion.

Stock-Level: determines the number of recently sold items that have a stock level

below a specified threshold. It represents a heavy read-only database transaction

with a low frequency of execution, a relaxed response time requirement, and

relaxed consistency requirements.

autonomic workload management for database ... - QSpace

Documents

Transcript of autonomic workload management for database ... - QSpace