Masaryk UniversityFaculty of Informatics

}w���������� ������������� !"#$%&'()+,-./012345<yA|Extend WildFly ApplicationServer logging capabilities

Diploma Thesis

Bc. Radek Koubský

Brno, 2016

Declaration

Hereby I declare, that this paper is my original authorial work, whichI have worked out by my own. All sources, references and literatureused or excerpted during elaboration of this work are properly citedand listed in complete reference to the due source.

Bc. Radek Koubský

Advisor: RNDr. Adam Rambousek, Ph.D.

ii

Acknowledgement

I would like to thank to my faculty supervisor RNDr. Adam Ram-bousek, Ph.D. for giving me valuable advices about writing this the-sis. I would also like to thanks to my supervisor from Red Hat Czech,s.r.o, Mgr. Martin Večeřa, for helping me with critical decisions dur-ing the development. Many thanks are due to Andrew Dinn, the au-thor of the Byteman, for his excellent technical and communicationskills during our cooperation.

iii

Abstract

The aim of this thesis is to extend Wildfly Application Server log-ging capabilities. The extension consists of two parts. The first partprovides storing log events to Elasticsearch in order to analyze andvisualize the log events using Kibana tool across multiple server in-stances (e.g. cluster or cloud installation). The second part uses theByteman project to inject new log events into selected parts of theWildfly bytecode. Thus, the implementation of new log events canbe used across different versions of Wildfly. Such parts include: WebContainer (Servlets, JSPs), REST endpoints, WS endpoints, EJB invocationsand JMS communication.

iv

Keywords

wildfly, application server, logging, bytecode manipulation, byteman,java, elasticsearch, kibana, jboss

v

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . 2

2 Wildfly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1 Structure of Wildfly . . . . . . . . . . . . . . . . . . . . . . 42.2 JBoss Modules . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Wildfly logging subsystem . . . . . . . . . . . . . . . . . . 8

2.3.1 Loggers . . . . . . . . . . . . . . . . . . . . . . . 92.3.2 Handlers . . . . . . . . . . . . . . . . . . . . . . . 102.3.3 Defining custom handler . . . . . . . . . . . . . 10

3 Byteman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.1 Introduction to Byteman . . . . . . . . . . . . . . . . . . . 123.2 The Byteman Rule Language . . . . . . . . . . . . . . . . . 14

3.2.1 Event Condition Action Rules . . . . . . . . . . . 143.2.2 Rule events . . . . . . . . . . . . . . . . . . . . . 153.2.3 Rule Bindings . . . . . . . . . . . . . . . . . . . . 163.2.4 Rule Expressions . . . . . . . . . . . . . . . . . . 173.2.5 Rule actions . . . . . . . . . . . . . . . . . . . . . 173.2.6 Built-In Calls . . . . . . . . . . . . . . . . . . . . 173.2.7 User-Defined Rule Helpers . . . . . . . . . . . . 17

3.3 Agent Transformation . . . . . . . . . . . . . . . . . . . . . 193.4 Rule engine . . . . . . . . . . . . . . . . . . . . . . . . . . 203.5 Using Byteman in Java applications . . . . . . . . . . . . . 21

3.5.1 Example of Byteman in Java application . . . . . 224 Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1 Introduction to Elasticsearch . . . . . . . . . . . . . . . . . 244.2 Basic concept . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.2.1 Cluster . . . . . . . . . . . . . . . . . . . . . . . . 254.2.2 Node . . . . . . . . . . . . . . . . . . . . . . . . . 254.2.3 Index . . . . . . . . . . . . . . . . . . . . . . . . . 254.2.4 Type . . . . . . . . . . . . . . . . . . . . . . . . . 254.2.5 Document . . . . . . . . . . . . . . . . . . . . . . 26

4.3 Accessing Elasticsearch . . . . . . . . . . . . . . . . . . . . 264.3.1 RESTful API . . . . . . . . . . . . . . . . . . . . . 264.3.2 Java API . . . . . . . . . . . . . . . . . . . . . . . 27

vi

Node client . . . . . . . . . . . . . . . . . . . . . 27Transport client . . . . . . . . . . . . . . . . . . . 28

5 Analysis and Design . . . . . . . . . . . . . . . . . . . . . . . 295.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 295.2 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.3 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.3.1 IDE . . . . . . . . . . . . . . . . . . . . . . . . . . 325.3.2 Maven . . . . . . . . . . . . . . . . . . . . . . . . 325.3.3 Git . . . . . . . . . . . . . . . . . . . . . . . . . . 335.3.4 Kibana . . . . . . . . . . . . . . . . . . . . . . . . 335.3.5 Bytecode manipulation tools . . . . . . . . . . . 33

6 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 366.1 Elasticsearch Wildfly Log . . . . . . . . . . . . . . . . . . . 366.2 Byteman Wildfly Log . . . . . . . . . . . . . . . . . . . . . 38

6.2.1 Log Helper class . . . . . . . . . . . . . . . . . . 386.2.2 Byteman rules . . . . . . . . . . . . . . . . . . . . 38

7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

vii

1 Introduction

Any serious software project needs to have a mechanism to recordthe state of the project at runtime, thus it is possible to trace its un-expected behavior. This requirement comes from idea that perfect,bug free software does not exist. For this purpose, the logging termwas introduced. In IT terminology, there are various definitions oflogging, one of them can be found in [1]: ”Logging is a systematicand controlled way of representing the state of an application in ahuman-readable fashion.” There is no doubt about benefits emerg-ing from logging, namely: problem diagnosis, quick debugging, cost andtime saving, easy maintanance and history.

The benefits of logging should lead to design and implement ap-plications with robust and comprehensive logging system. However,in the context of Open Source Software, there is a lack of logging in-formation in many Open Source projects from particular reasons. Forexample, in the area of applications servers, one of these reasons canbe performance requirements. The demand for a high performanceapplication server requires fast and compact implementation, thusany additional log information is undesirable.

Solution for insufficient logging information is to extend existinglogging of a target application. Nevertheless, providing such exten-sion is not trivial as it depends on many factors and requires deepanalysis of the target application. One of the main problems of log-ging extension is different motivation between users of the applica-tion and developers. Whereas users require additional logging to beimplemented, developers has lack of motivation to add new logginginformation to the source code. Thus, the only solution is to createseparate branch of the existing source code in order to implementlogging extension. However, this approach has clear disadvantage asthe extension is valid only for specific version of an application.

This work focuses on extension of Wildly Application Server log-ging capabilities. Wildfly is Open Source project with strong back-ground in the JBoss Community1. The server evolves fast in a short pe-riod of time to reflect requirements from the community. Thus, newversions of Wildfly are continuously released.

1. JBoss Community is a community of open source projects.

1

1. Introduction

1.1 Goals

The main goal of this thesis is to extend Wildly Application Serverlogging capabilities. This goal can be split into three subgoals whichtogether forms the logging extensions.

The first goal is to study Wildfly logging subsystem and designthe first part of the extension; storing log events to Elasticsearch. Wild-fly defines best practices in order to extend the logging subsystem,thus the implementation follows straightforwards steps designed forthe extension.

The second goal is to visualize log events using Kibana tool. Itshould be possible to diligently monitor all important events acrossmultiple server instances (e.g. cluster or cloud installation).

The third and the most important goal is to analyze selected partsof the Wildlfy source code and identify missing logging information.Then, an appropriate methods (e.g. invocation handlers, AOP2, reflec-tion, bytecode manipulation) must be proposed and implemented.Selected parts of Wildfly must include: Web Container (Servlets, JSPs),REST endpoints, WS endpoints, EJB invocations and JMS communication.

1.2 Structure of the Thesis

The following text outlines individual chapters as they form the struc-ture of this thesis.

The chapter Wildfly focuses on the Wildfly Application Server. Itemphasizes on the architecture of the Wildfly core and logging sub-system. At the end of the chapter, the example of custom handler ex-tension is described as it is used in the implementation part.

The chapter Byteman describes Byteman, a bytecode manipula-tion tool which was selected for manipulation with Wildfly bytecode.Detail information about Byteman and its functionality is providedwith simple example of using Byteman in Java application.

The Elasticsearch chapter provides basic concept of the Elastic-search technology in order to understand its role as a store of the logevents from Wildfly.

After knowledge about the key concepts of this work, the chapter

2. AOP - Aspect-oriented programming

2

1. Introduction

Analysis and Design focuses on requirements analysis, design andused tools including analysis of bytecode manipulation tools.

The chapter Implementation shows technical details about the im-plementation, which is developed with best practices and emphasison extensibility, maintainability and readability of the source code.

The last chapter Conclusion summarizes achieved goals, prob-lems emerged during the implementation and future work.

3

2 Wildfly

In this chapter, the Wildfly Application Server (former JBoss AS) is de-scribed. Because of the server complexity, only specific part are men-tioned and description of its internal structure and configuration isprovided. Further information about Wildfly can be found at officialwebsite http://wildfly.org/. Wildfly 8 Documetation [2] is used asthe main source for this chapter.

Wildfly is one of the leading application servers in the area of en-terprise solutions. From [2, Getting Started with WildFly 8]: ”WildFly8 is an exceptionally fast, lightweight and powerful implementationof the Java Enterprise Edition 7 Platform specifications.” The Wildflyproject is licensed as Open Source and supported by the JBoss Commu-nity. However, there is commercial version of Wildfly, JBoss EnterpriseApplication Platform, which is licensed and supported by Red Hat.

2.1 Structure of Wildfly

The structure of Wildfly has been changed in recent years due to re-quirements on maintainability and performance. Wildfly is formedby the wildfly core and the set of subsystems. Subsystems are definedas on-demand services which are called by applications deployed onthe server. Each subsystem is responsible for specific functionality.

The core of the server was significantly changed in the versionJBoss AS 7. It consists of two parts [3]:

• JBoss Modules: ”This handles the class loading of resources inthe container. You can think of JBoss Modules as a thin boot-strap wrapper for executing an application in a modular envi-ronment.”

• Modular Service Container (MSC): ”This provides a way toinstall, uninstall, and manage the services used by a container.MSC further enables the injection of resources into services anddependency management between services.”

The structure of the core described above has remained to the last

4

2. Wildfly

version of Wildfly. The following diagram illustrates basic architec-ture of the Wildfly core:

Figure 2.1: Wildfly core architecture [4]

2.2 JBoss Modules

Class loading is a problematic domain in the context of applicationservers. Wildfly comes with the JBoss Modules class loading systemwhich provides an effective way of loading classes into a class path.JBoss Modules is defined as follows [5, Introduction]: ”JBoss Modulesis a standalone implementation of a modular (non-hierarchical) classloading and execution environment for Java.”

5

2. Wildfly

JBoss Modules is implemented as a delegating class loader modelwhich is thread-safe, fast, and highly concurrent. Each JAR which isloaded into the class path is handled as a module. A module mustdefine a list of dependencies only to modules it depends on. Modulesare loaded only on a request by an application, thus the performanceof the application is dependent on modules which are currently usedby the application.

Wildfly defines two types of modules [4]:

• Static modules

• Dynamic modules

The following diagram shows the architecture of the static anddynamic modules:

Figure 2.2: JBoss modules architecture [4]

Static modules are defined in form of a hierarchical directory struc-ture. In Wildfly 8, static modules are located within the JBOSS_HOM-E/modules/system/layers/base directory. A directory structure of a mod-ule is derived from the name and version of the module. The specificdirectory of the module contains a module descriptor and resourcesrequired by the module (JAR files).

6

2. Wildfly

The module descriptor is represented by the module.xml file. Thefollowing example shows the module descriptor of the javax.batch.apimodule.xml module:<module xmlns="urn:jboss:module:1 .3" name="javax

.batch.api"><resources >

<resource -root path="jboss -batch -api_1.0_spec -1.0.0. Final.jar"/>

</resources ><dependencies >

<module name="javax.api"/><module name="javax.enterprise.api"/>

</dependencies ></module >

Listing 2.1: An example of a module descriptor [4]The root <module> element defines resources and dependencies of themodule.

Resources are defined within the <resources> element which con-tains one or more resource-root> elements. Each <resource-root>element has the path attribute which specifies a path to the JAR file.The example above uses only one resource, the jboss-batch-api_1.0_spec-1.0.0.Final.jar file located in the module directory next to the mod-ule.xml file.

The next element, <dependencies>, is used to define dependen-cies for a module. A dependency is defined by the <module> elementwhich identifies specific module in the hierarchical directory struc-ture. In the example above, javax.api and javax.enterprise.api moduleswill be resolved as dependencies for the javax.batch.api module.

Dynamic modules are defined for each deployment of an applica-tion. There are two approaches how to create dynamic module [4].

First option is to define dependencies within a file called MAN-IFEST which is placed in a JAR file of an application. Dependenciesof the JAR file are defined in the Dependencies header; each depen-dency is separated by comma. The following line is an example of aMANIFEST file with one dependency:Dependencies: org.jboss.logging

7

2. Wildfly

The second option is to configure the deployment-structure.xml fileby adding module dependency to it:<jboss -deployment -structure >

<deployment ><dependencies >

<module name="org.jboss.logging"/></dependencies >

</deployment ></jboss -deployment -structure >

Listing 2.2: Configuring the deployment structure file [4]

2.3 Wildfly logging subsystem

The logging subsystem is responsible for displaying all logs whichappear in the application server i.e. in every subsystem and user ap-plications. The logging subsystem consists of the following parts: han-dler configurations, logger, the root logger declarations and logging profiles.The loggers declare a set of handlers which send logs to defined des-tinations such as a console, a file, a network and others [2, LoggingConfiguration].

The logging configuration can be defined in XML files (e.g. stan-dalone.xml, domain.xml etc.), a web console or a command line inter-face. The following example illustrates a snippet of the logging sub-system configuration from an XML file:<subsystem xmlns="urn:jboss:domain:logging:1 .0">

<console -handler name="CONSOLE"><level name="DEBUG"/><formatter >

<pattern -formatter pattern="%d{HH:mm:ss ,SSS} %-5p [%c] (%t) %s%E%n"/>

</formatter ></console -handler ><logger category="com.arjuna">

<level name="WARN"/></logger ><root -logger >

8

2. Wildfly

<level name="DEBUG"/><handlers >

<handler name="CONSOLE"/></handlers >

</root -logger ></subsystem >

Listing 2.3: Configuring the logging subsystem

The <console-handler> element identifies a handler which prints logevents from Wildfly to a console. Handler element contains two childelements, the <level> and the <formatter> element. The level ele-ment specifies a log level the outgoing logs must match. The format-ter element defines output format for logs in the console.

The <logger> element defines a logger and its level for specificpackage or class. Target package or class is specified by the categoryattribute.

The <root-logger> element is same as the logger element only itdoes not have the category attribute as it is the root logger.

2.3.1 Loggers

Log messages are created by loggers. Definition of a logger consists ofa category which is represented by package or class name. Whenevera logger tries to log a message, it first checks a level of the message.If the level of the message is greater than the level of the logger, afilter is applied to the log message. A filter determines whether a logmessage is loggable or not according to the filter rule. The loggingsubsystem provides rich set of rules (expressions) for filters; detaileddescription of all rules can be found in the Logging documentation[2, Logging Configuration].

Wildfly logging subsystem defines a set of logger levels [2, Log-ging Configuration], the most known examples:

• TRACE

• DEBUG

• FINE

• INFO

• WARN

• ERROR

• SEVERE

• OFF

9

2. Wildfly

2.3.2 Handlers

Handlers represent next step in the log processing. If a log messageis loggable, a handler processes the message e.g. sends the messageto a system console or stores the message to a file.

The logging subsystem provides the following set of handlers [2]:

• async-handler

• console-handler

• file-handler

• periodic-rotating-file-handler

• size-rotating-file-handler

• syslog-handler

However, a custom handler can be implemented for user specific re-quirements.

2.3.3 Defining custom handler

The logging subsystem provides straightforward process of a cus-tom handler implementation. The custom handler represents stan-dard module within the JBoss Modules architecture 2.2. Thus, thehandler as a module can benefit from the module system, for exampledefining dependencies to other modules, version management etc.

The simplest way to create a custom handler is to extend the java.u-til.logging.Handler abstract class which is a part of Java Logging API.The logging API consists of several classes that provide manipula-tion with logging information within an Java application. The Wildflylogging subsystem uses the abstract handler class and its implemen-tations to process all log events within Wildfly.

10

2. Wildfly

The following example shows the custom handler class MyHan-dler.java:package com.examples.loggingpublic class MyHandler extends Handler {

@Overridepublic void publish(LogRecord record) {

//do something with log}

@Overridepublic void flush() ...@Overridepublic void close() ...

}

Listing 2.4: Custom handler class

The next step in custom handler implementation is to create the mod-ule.xml file with proper configuration according to 2.1. Then the my-Handler.jar with the module.xml file is placed to a specific module di-rectory, for example JBOSS_HOME/modules/system/layers/base/com/ex-amples/myhandler/main. The final step of the custom handler imple-mentation is to add the custom handler to the logging subsystem con-figuration:<custom -handler name="MyHandler" class="com.

examples.logging.MyHandler" module="com.examples.myhandler"><level name="DEBUG"/><formatter >...</formatter >

</custom -handler >

Listing 2.5: Adding custom handler to the logging subsystemconfiguration

After all previous steps, the MyHandler will be used to process logevents in Wildfly.

11

3 Byteman

This chapter aims to introduce Byteman, a bytecode manipulation tool,and its use in Java applications. Fundamental principles, functions,and structure of Byteman are described. At the end of the chapter,an example of simple java application is provided in order to un-derstand bytecode injection which Byteman uses. More detailed doc-umentation, tutorials, and guidelines which cover almost every as-pect of Byteman can be found in different versions at official websitehttp://byteman.jboss.org/. Primary source for this thesis is Byte-man documentation 3.0.0 [6].

3.1 Introduction to Byteman

Byteman is a bytecode manipulation tool which uses bytecode injec-tion1 in order to change a java application as it is being loaded by Javavirtual machine or while the application is running [6, Introduction toByteman].

Because of the bytecode injection, there is no need to change theoriginal source code nor to recompile or redeploy the application it-self. In other words, it is possible to add, replace or remove injectedcode at runtime without interrupting a Java application. Bytemanalso provides a way to inject code into the Java language by modifyingclasses such as java.lang.Thread and other classes from java.lang pack-age. Byteman uses simple scripting language called Event ConditionAction (3.2.1) rules to determine how the original Java code should betransformed according to event and condition.

The original purpose of Byteman was to provide a mechanism forautomation tests used in multi-threaded and multi-JVM Java appli-cations using a technique called fault injection. Byteman can be usedto inject faults which cause an application to invoke unusual or unex-pected methods which are required for certain test scenarios. Thereare four areas supported by Byteman in automation testing [6, Intro-duction to Byteman]:

1. Bytecode injection (or bytecode instrumentation) is a process of modifying theactual bytecode of a class.

12

3. Byteman

• tracing execution of specific code paths and displaying applica-tion or JVM state

• subverting normal execution by changing state, making unsche-duled method calls or forcing an unexpected return or throw

• orchestrating the timing of activities performed by independentapplication threads

• monitoring and gathering statistics summarising application a-nd JVM operation

Apart from the testing area, the concept of bytecode injection involvesByteman in much larger scope of use. The rule engine 3.4, whichforms the core of Byteman, is able to inject bytecode into almost everyplace in a call stack of a running java application. Thus the capabilityof bytecode injection enables manipulation of a Java program with-out any limitations in the design of the language (e.g. access levelmodifiers).

Byteman offers a suite of built-in operations and conditions in or-der to support activities and tasks emphasized above. Built-in opera-tions (such as delays, waits, countdowns, flag operations and signals)are mainly used for coordination of threads in multi-threaded appli-cations where multiple threads run with no order and it is needed tocontrol the flow of particular threads.

To simplify process of using Byteman in common use or in the testautomation, the Byteman rulecheck plugin and the BMUnit projectswere created in later releases of Byteman. Byteman rulecheck plugin2

is a maven plugin which consists of a rule parser and a type checker.It runs the rule parser and type checker before every execution of anymaven test to ensure that rules are still valid when a test code is modi-fied. BMUnit3 is a java library which provides integration of Bytemanwith the two most used test frameworks, JUnit4 and TestNG5. BMUnit

2. https://developer.jboss.org/wiki/CheckingYourBytemanRuleScriptsUnd-erMaven3. https://developer.jboss.org/wiki/BMUnitUsingBytemanWithJUnitOrTest-NGFromMavenAndAnt4. http://junit.org/5. http://testng.org

13

3. Byteman

uses java annotations to load Byteman agent into JVM during tests,and to specify which rules should be installed by the agent before atest run (and uninstalled after test is completed).

Byteman is an Open Source project and plays important role inJBoss Community. It helps other JBoss projects with test automationand quality assurance.

3.2 The Byteman Rule Language

This section introduces fundamentals of the Byteman Rule Language,its syntax, semantics and functionality. The Byteman Rule Languageis the only way how to inject bytecode into Java applications via Byte-man. Understanding of the Byteman Rule Language is required in or-der to write Byteman rule scripts which form the core functionality ofthis thesis. The following text is mainly focused on such parts of theByteman Rule Language which are used in this work. Full descriptionof the Byteman Rule Language is included in Byteman documenta-tion [6, The Byteman Rule Language].

3.2.1 Event Condition Action Rules

According to [7]: ”Event Condition Action (ECA) languages are anintuitive and powerful paradigm for programming reactive systems.”

The Byteman Rule Language is built on the Event Condition Ac-tion language and adopts its core features. Byteman employs ECArule engine to inject bytecode at specified location during execution.The whole process of the bytecode injection is done by strict and un-ambiguous steps which arise from the ECA language.

A set of rules is defined in Byteman script which is a file witha .btm suffix. Rules are used to determine how a specific part of anapplication should be transformed at runtime. A rule comprises threecomponents, event, condition and action, which have the followingsemantics[7]:

1. when Event occurs

2. if Condition is verified

3. then execute Action

14

3. Byteman

The ECA formalism provides clear and easy-to-understand ap-proach for performing bytecode injection without ambiguity.

Syntax of the rule consists of several parts which can be parsed bythe rule engine. The following code snippet illustrates rule schemeand its structure:# rule skeletonRULE <rule name># comment line in rule bodyCLASS <class name>METHOD <method name>BIND <bindings >IF <condition >DO <actions >ENDRULE

Listing 3.1: Byteman rule scheme [6]

A rule starts with a name placed after the RULE keyword. Thename can have any form of a character sequence which length is atleast one non-white space character. Rule names can be duplicated,however, it causes ambiguity in debugging rule scripts. The rule nameis the only identifier if an error occurs while parsing, type checking,compilation or execution of the rule.

3.2.2 Rule events

A rule event is an event related to an event part of the ECA languagespecification 3.2.1. Byteman specifies the rule event as a location in atarget method within a target class [6, Rule Events]. There are threetypes of target method: a static method, an instance method and a con-structor.

Target class and target method are defined after the CLASS andMETHOD keywords and must be in defined order on separate lines.The class name describes a class with or without the package specifi-cation. The method name can be defined either as a plain name or asa name with arguments and return type. The constructor method isdefined by the special string <init> in the method name.

The rule event can employ any class of a Java application exceptthe following classes:

15

3. Byteman

• classes in package java.lang

• classes belonging to the Bytemam itself (org.jboss.byteman pack-age)

Injecting into Java classes is not allowed by default. Nevertheless,this option can be disabled by configuring the Java agent (see [6, Avail-able -javaagent Options]).

Byteman includes a set of location specifiers which are describedin [6, Location Specifiers], the following list of location specifiers isused in this work:

• AT ENTRY

• AT EXIT

• AT LINE number

• AFTER WRITE $var-or-idx [count | ALL ]

• AT INVOKE [ type .] method [ ( argtypes ) ] [count | ALL ]

• AFTER INVOKE [ type .] method [ ( argtypes ) ][count | ALL ]

Location specifier must be placed after the METHOD keyword.If location specifier is omitted, the rule engine uses the AT ENTRYlocation specifier.

3.2.3 Rule Bindings

The rule binding specification is used for binding values to variablesin a context of a rule. It is possible to assign a value to a new variable orreassign an existing variable. The context of the rule provides accessto methods and fields of target class instance (if the target method isnot static) and local variables in the scope of the trigger method. Theassignments are performed each time the rule is triggered [6, RuleBindings].

16

3. Byteman

3.2.4 Rule Expressions

Rule expressions can be used in the rule body and provide severaloperations such as method invocations, built-in calls and field refer-ences. The rule expressions are mostly composed by the $ symbol fol-lowed by reserved words, for example $0, $CLASS, $METHOD. It ispossible to combine simple expressions to construct complex expres-sions using standard Java operators. The full set of rule expressionsis described in [6, Rule Expressions].

3.2.5 Rule actions

Rule actions can be represented by rule expressions 3.2.4 or return orthrow actions. A rule expression which is used in an action may haveany type including void type [6, Rule actions]. Several rule expres-sions are separated by semicolon. However, if an action contains onlya single rule expression, the semicolon is omitted.

A rule expression can employ standard built-in methods (3.2.6) ormethods defined in additional helper class (3.2.7) bound to a rule.

3.2.6 Built-In Calls

Built-in calls are operations which can be used within a rule body.Byteman provides a set of build-in calls in order to write comprehen-sive rules without additional complexity [6, Built-In Calls].

Build-in calls are implemented as public methods of the defaultHelper class. An instance of the Helper class is assigned to a rule in-stance every time an instance is being created by the rule engine. Thusa rule can use all public methods exposed by the instance of the Helperclass. Full list of built-in operations and its description can be foundin the documentation of Byteman [6].

3.2.7 User-Defined Rule Helpers

Built-in operations in the previous subsection provide default func-tionality for a user of Byteman. However, it is possible to define aCustomHelper class which can override, replace or extend the existingdefault Helper class [6, User-Defined Rule Helpers].

17

3. Byteman

The only requirement for the CustomHelper is to extend the baseHelper class. The CustomHelper automatically inherits all built-in meth-ods from the base Helper if the methods are not overridden. A newmethod in the CustomHelper must have public access level modifier.Thus, a rule can use methods exposed by the CustomHelper.

The rule in the following example employs helper class MyHelper:

RULE rule with helperCLASS org.jboss.as.ejb3.security.

SecurityContextInterceptorMETHOD processInvocation(InterceptorContext)HELPER com.examples.MyHelperAT ENTRYIF trueDO myMethod ();ENDRULE

Listing 3.2: Rule with user-defined helper class

When the rule above is executed, an instance of MyHelper classis created and method myMethod() is invoked in the rule action. My-Helper class is defined as standard Java class:

public class MyHelper extends Helper{protected MyHelper(final Rule rule) {

super(rule);}public String myMethod (){

System.out.println("Printed text fromMyHelper class.");

}}

Listing 3.3: User-Defined helper class

A CustomHelper can be bound either to a single rule or to all rules(without specific helper class) defined in a rule script. Custom helpersare identified by the full package qualified class definition.

18

3. Byteman

3.3 Agent Transformation

Byteman employs Java agent to modify bytecode of the target Javaapplication. From [8]: ”An agent is anything that can be viewed asperceiving its environment through sensors and acting upon that en-vironment through effectors.”

The Java agent uses package java.lang.Instrumentation for bytecodeinstrumentation [9]: ”Bytecode instrumentation is a valuable tech-nique for transparently enhancing virtual execution environments,for purposes such as monitoring or profiling.”

The Byteman agent and rule scripts are loaded at JVM boostrap.When an application is launched, the agent analyzes application byte-code and looks for trigger points which are defined by the rule events3.2.2. If a rule event matches a location in bytecode, the agent insertstrigger call into that location. A trigger call is performed by the ruleengine and consists of the following parts [6, Agent Transformation]:

• the trigger method, i.e. the method which contains the triggerpoint

• the rule which has been matched

• the arguments to the trigger method

If there is more than one rule for the same trigger point, a sequenceof trigger calls will be added to the bytecode. The order of trigger callsreflects the order of rules defined in a rule script. A special case isrules with an AFTER location specifier 3.2.2. In this case, trigger callsare executed in reverse order in which rules were defined in a rulescript.

When a program flow reaches a location with a trigger call, therule engine searches for the rule which is bound to the trigger call andexecutes it. At the start of the rule execution, the rule engine initializesvariables defined in rule bindings 3.2.3. Then, the execution continuesin the ECA formalism 3.2.1: if the rule condition is verified, the ruleaction is executed.

The rule engine obtains an instance of a target method and methodarguments from a trigger call. To access the instance and method ar-guments in a rule body, $0 (the instance) and $1, $2... (method argu-ments) expressions are used 3.2.4. Local variables are also passed by

19

3. Byteman

the trigger call and can be accessed with $ symbol followed by a nameof a local variable e.g. $variableName.

The agent also provides exception handler which is wrapped aro-und the trigger call in order to process exceptions which can be thrownduring the execution of a rule [6, Agent Transformation].

3.4 Rule engine

Byteman rule engine is heart of the Byteman, it executes rules to in-ject bytecode into Java application.. The following components formthe Byteman rule engine: a rule parser, type checker and interpreter/-compiler. The agent invokes the rule parser at JVM bootstrap in orderto search for potential trigger points [6, ECA Rule Engine].

When a rule is type checked, the type checker needs to accessproperties of the target class and all classes (including fields, methodsignatures etc.) which the target class employs. For this reason, therule must be type checked and compiled after the corresponding byte-code of the target class was loaded i.e. at the first time the rule wastriggered. A positive side effect of the type checking process is reduc-tion of cost as rules, which are not triggered, are not type checked andcompiled.

Depending on the event specification, there can be one or moretrigger points associated with a rule. However, if the rule specificationidentifies exactly one trigger point for the target class, the class can beloaded by different class loaders. Thus, the rule engine ensures thata rule is type checked for each trigger point.

If there is an error during type check or compilation, the rule en-gine informs user about the error and the trigger calls will not be exe-cuted.

The rule engine uses helper adapter, an auxiliary class, for the ex-ecution of a rule. The helper adapter extends the Helper class and pro-vides additional functionality for the execution. The execute method,which is invoked at the trigger point, is responsible for the rule exe-cution itself.

When a rule is executed, a instance of the helper adapter is created.The instance initializes the rule with all bindings fields to providemetadata for the trigger call. Then, the rule engine calls the execute

20

3. Byteman

method to execute the rule.A new instance of helper adapter is created for each invocation of

same rule, thus, undesirable interference of concurrent trigger calls isavoided.

3.5 Using Byteman in Java applications

Using Byteman in a Java application means running the Java agentin the background of an application. For this purpose, the -javaagentoption must be passed to the Java virtual machine which runs the Javaapplication. The Java agent is configured in the following format [6,Running Applications with Byteman]:-javaagent:path_to_Byteman.jar=option1 , option2 ,

...

Byteman agent provides a set of agent options for different purposes,the following list shows several important options used in this work:

• script:scriptfile

• sys:jarfile

• boot:jarfile

The script:scriptfile defines Byteman rule script file which is loadedat JVM startup. The scriptfile is the name of the file. Byteman agentenables multiple script:scriptfile options in order to load more thanone rule script.

The sys:jarfile option provides loading a .jar file whose classes areavailable on the system class path. This ensures that classes in thejar file can be found when rules are executed by the rule engine. Thejarfile is path to existing jar file. This option is typically used for user-defined Helper classes to be available in rules.

The boot:jarfile option is similar to the sys:jarfile. Only difference isavailability of classes in the .jar file on the bootstrap class path. This isrequired when bytecode is injected into classes from java.lang packageand other JVM classes.

The full description of all agent options can be found at [6, Avail-able -javaagent Options].

21

3. Byteman

3.5.1 Example of Byteman in Java application

The following definition of a java class represents simple Java appli-cation which will be used to show the Byteman basic functionality:package com.examples;public class BytemanTest{

public static void main(String [] args){System.out.println("Inside the

BytemanTest.main");}

}

Listing 3.4: Java class bypassed by the Byteman

The class above contains only main method which prints text "Insidethe BytemanTest.main" to system output.

The next step is to create Byteman rule script script.btm and defini-tion of a rule which is responsible for injecting bytecode into the classabove. The following rule is defined for the BytemanTest class and itsmain method:RULE testBytemanCLASS com.examples.BytemanTestMETHOD mainAT ENTRYIF trueDO debug("Main method bypassed by Byteman.");ENDRULE

Listing 3.5: Example of Byteman rule

To run the application with Byteman, the -javaagent option must beplaced before launching the com.examples.BytemanTest application. Thefollowing example runs the application in Linux command line:> java -javaagent: ${ BYTEMAN_HOME }/lib/byteman.

jar=script:script.btm com.examples.BytemanTest

22

3. Byteman

Running the application with the command on previous page pro-duces the following output in command line:> java -javaagent: ${ BYTEMAN_HOME }...> Main method bypassed by Byteman.> Inside the BytemanTest.main

23

4 Elasticsearch

This chapter briefly describes Elasticsearch technology which is usedto store and search data. In the context of this work, the data repre-sents log messages from Wildfly Application Server 2.

4.1 Introduction to Elasticsearch

Elasticsearch is mainly known as a search engine. From [10]: ”Elastic-search is an open-source search engine built on top of Apache Lucene,a fulltext search-engine library.”

It is an abstract layer between complex Apache Lucene1 library anduser applications. Elasticsearch does not require for its users to fullyunderstand the complexity of Lucene, it provides API for workingwith Lucene in simple way.

However, Elasticsearch is not only search engine, it can be definedas follows [10]:

• A distributed real-time document store where every field is in-dexed and searchable

• A distributed search engine with real-time analytics

• Capable of scaling to hundreds of servers and petabytes of struc-tured and unstructured data

Elasticsearch is implemented in Java and packaged into standaloneserver. Access to the Elasticsearch server can be done in various ways,such as RESTful API2, web client (Java, .NET, Python etc.) or com-mand line interface.

1. Lucene is arguably the most advanced, high-performance, and fully featuredsearch engine library in existence today—both open source and proprietary [10].2. Representational State Transfer (REST) is architectural style, developed as anabstract model of the Web architecture [11].

24

4. Elasticsearch

4.2 Basic concept

Several concepts form the fundamentals of the Elasticsearch core, thusunderstanding of these concepts is required before using Elasticsearchin practice [12].

4.2.1 Cluster

A Cluster consists of one or more nodes (servers) that together storeall data. Unified indexing and search operations are performed acrossall nodes in a cluster. Every cluster has its own name which is unique;default value of the name is elasticsearch.

4.2.2 Node

A Node represents a server within a cluster. It is responsible for store,indexing and search operations in the context of a cluster that it be-longs to. A node is identified by name. The name is used to configurewhich server in network is related to particular node in a cluster.

A node joins a cluster by specifying the cluster name. If the clus-ter name is not defined, nodes are configured to join the cluster withname elasticsearch by default. Elasticsearch does not define anylimitations for the number of nodes in a single cluster.

4.2.3 Index

A set of documents with similar attributes can be bound to an In-dex. Whenever an operation is performed on a document, the indexrelated to the document is employed.

An index is defined as a string containing only lowercase charac-ters. The number of indexes in a cluster is not limited.

4.2.4 Type

A Type is another way of data partitioning within an index. Depend-ing on an user requirements, one or more types can be defined foran index. A type is usually used for documents which have commonfields.

25

4. Elasticsearch

4.2.5 Document

Data in Elasticsearch are stored as a Document. Once a document iscreated under an index, data within a document are searchable in acluster. A document is represented in JSON (JavaScript Object Nota-tion) which is well known data interchange format.

Although a document is related to an index, it must be also boundto a type within an index.

4.3 Accessing Elasticsearch

Connecting to Elasticsearch cluster can be done in several ways de-pending on user requirements. There are two main approaches forworking with Elasticsearch: RESTful API and Java API [10, Talking toElasticsearch].

4.3.1 RESTful API

Elasticsearch provides comprehensive RESTful API which uses HTTPprotocol and JSON as a data format, the default port for communica-tion is 9200. The RESTful API is used by the command line interfaceand web clients.

The command line interface is the basic technique for communi-cation with Elasticsearch. It uses cURL3 for making HTTP requestsand displaying content returned in responses.

The web clients are built on the RESTful API and provides accessto Elasticsearch in various languages such as .NET, JavaScript, Pythonand many others.

An Elasticsearch request is similar to an HTTP request, it consistsof the following parts [10, Talking to Elasticsearch]:

curl -X<VERB> '<PROTOCOL >://<HOST >:<PORT >/<PATH>?<QUERY_STRING >' -d '<BODY >'

For example, the following command creates a document with id1 which belongs to bookstore index and order type:

3. cURL is a computer software project providing a library and command-line toolfor transferring data using various protocols.

26

4. Elasticsearch

curl -XPUT 'http:// localhost:9200/bookstore/order/1' -d '

{"number": "20150001" ,"price": "2015 -01 -15"}'

Listing 4.1: Example of indexing data in Elasticsearch

The same document can be retrieved by the GET method:curl -XGET 'http:// localhost:9200/bookstore/

order /1? pretty '{"number": "20150001","price": "2015 -01 -15"}

Listing 4.2: Example of retrieving data from Elasticsearch

4.3.2 Java API

The Java API is the main API used for interaction with Elasticsearch.It directly communicates with Elasticsearch cluster using the nativeElasticsearch transport protocol over port 9300. Note that nodes within acluster also use port 9300 for communication in the cluster [10, Talk-ing to Elasticsearch]. The Java API provides fully asynchronous envi-ronment, thus all operations are performed parallelly. In addition, theBulk API can be used to aggregate and execute multiple client oper-ations e.g. index several documents in a single request [13, Java API].The core of all other APIs [12] is formed by the Java API which in-ternally executes requests from the APIs. The Java API provides twobuilt-in clients, the Node client and the Transport client.

Node client

The Node client operates as a single node within a cluster, thus it mustjoin the cluster to be a part of it. However, it does not hold any data,it only redirects all requests to proper node in a cluster. To find the

27

4. Elasticsearch

corresponding node which holds requested data, the Node client ad-ministrates information about every node and its data in a cluster.Thus, it is able to forward a request directly to the correct node with-out performing a "double hop".

Transport client

The Transport client acts as a standard client in the client-server parad-igm. It connects to a remote cluster using one or more transport ad-dresses. Request which are sent over the transport addresses are pro-cessed in the round robin principle, thus, "double hop" is performedmost of the time.

28

5 Analysis and Design

This chapter covers requirements analysis and design of the extendedWildfly logging capabilities. Wildfly clearly defines the architectureof the logging subsystem which is described in 2.3. Thus the designof the first part of this work follows the steps provided by the loggingframework. On the contrary, the analysis of the second part focuses onexisting techniques which can be used to add log messages to Wildflywithout modifying the source code of the server itself.

5.1 Requirements

The main requirement is to extend Wildfly application server log-ging capabilities. The outcome of such extension must include anability to store log events to Elasticsearch, visualize log events usingKibana1 tool and add new log events to selected parts of Wildfly with-out changing Wildfly source code. This requirement can be split intothe following requirements:

Functional requirements

• Store log events to ElasticsearchIn order to visualize log events using Kibana, log events mustbe stored in an Elasticsearch cluster.

• Visualize log events using KibanaAs log events are persisted in Elasticsearch cluster, Kibana isemployed to visualize the log events.

• Add new log events to the selected parts of WildflySelected parts of Wildfly must be analyzed and new log eventsshould be added as needed. Such parts must include:

– Web Container (Servlets, JSPs)– REST endpoints

1. Kibana is an open source analytics and visualization platform designed to workwith Elasticsearch [14].

29

5. Analysis and Design

– WS endpoints– EJB invocations– JMS communication

Non-functional requirements

• Add new log events without modifying Wildfly source codeNew log events must be added and the source code of Wildflycannot be changed. Thus, the implementation is not dependenton a specific version of Wildfly.

5.2 Design

The extension of the Wildfly logging capabilities is designed in orderto reflects functional requirements and provides maintainable, exten-sible and simple solution for end users. The extension consists of twoparts, the Elasticsearch extension and the Byteman extension. Eachpart provides separate functionality. Note that a user does not needto use both extensions. For instance, there may be requirement onlyto store log events to the Elasticsearch cluster. Thus, the architecturemust be decoupled into independent components, each responsiblefor performing particular functionality. The architecture consists of 3main components: Elasticsearch, Kibana and Byteman.

The Elasticsearch component is represented by the Elasticsearchhandler, which is implemented as a custom logging handler describedin 2.3.3. The Elasticsearch handler is responsible for sending all logevents from Wildfly to an Elasticsearch cluster. The handler must beable to process large amount of log events and sends them to the clus-ter. In this case, Wildfly provides the async-handler (2.3.2) which canbe used for asynchron-ous processing of log events.

After storing log events to the Elasticsearch cluster, Kibana is em-ployed to visualize the log events. In the context of this work, Kibanarepresents standalone system which is separated from the architec-ture of the logging extension as it does not require any additionalimplementation.

The Byteman component is responsible for adding log events toWildfly without modifying its original source code. When a new log

30

5. Analysis and Design

event is added to Wildfly, it is processed by a set of handlers includingthe Elasticsearh handler. Thus, the new log event is immediately sentto an Elasticsearch cluster.

The following component diagram depicts the architecture of thelogging extension:

<<component>>Wildfly

<<component>>Elasticsearch

handler

<<component>>Byteman

<<component>>Elasticsearch

<<component>>Kibana

Java API

RESTful APIByteman rules

Powered ByVisual Paradigm Community Edition

Figure 5.1: The component diagram of the logging extension

5.3 Tools

This section introduces a set of tools and technologies used for theimplementation of the logging extension. Byteman and Elasticsearchare not mentioned in the following text as they are described in chap-ters 3 and 4. According to the nature of the Wildfly project, all toolsand technologies should meet the following requirements:

• Wildfly is Open Source, thus only Open Source technologies sho-uld be used for the implementation

• Wildfly is implemented in the Java language, this fact must betaken in consideration when choosing appropriate technologies

31

5. Analysis and Design

5.3.1 IDE

Effective development of any project should use an Integration De-velopment Environment. It consists of several tools which help de-veloper with fast and effective development. Such tools include a texteditor, a debugger, a build system, a version control system and oth-ers. Eclipse is free Open Source IDE for Java. It uses plugins for in-tegration with technologies such as Maven, Git, Wildfly and manyJBoss projects. In addition, Red Hat JBoss Developer Studio can beused as it is built on top of Eclipse and includes plugins for JBosstechnologies by default.

5.3.2 Maven

Development of an Java application requires an automation processfor building the source code of the application. Maven is a leadingbuild automation tool for Java based projects. This work uses severalaspects of Maven which are described in the following text.

To build a project with Maven, a project must define its projectobject model and a set of plugins which are shared across all projectsusing Maven. The project object model is defined by the pom.xml filewhich is a part of every Maven project [15].

A Maven project must define its unique identification which con-sists of: groupId, artifactId and version. This identification is definedwithin the pom.xml file which also contains all configuration proper-ties e.g. packaging (default is JAR), a project name, a description andany custom properties defined by a user.

One of the most important elements in Maven are dependencies. Itprovides comprehensive administration of libraries used in a Mavenproject. Libraries represents Maven projects which were published ina Maven repository. A dependency is defined by its unique identi-fication (groupId, artifactId, version) and a dependency scope2. De-pendencies are downloaded from a remote Maven repository duringa build process of a Maven project.

Maven repositories are defined in the settings.xml file which is cre-ated after installing Maven on a user machine. Maven provides accessonly to Maven Central Repository by default, thus other repositories

2. Dependency Scope defines certain limitations when processing a dependency.

32

5. Analysis and Design

must be specified within the settings.xml. In addition, a user can cre-ate its own repository which can be added to the settings.xml file asany other public repository.

5.3.3 Git

A version control system (CVS) provides necessary functions for soft-ware development such as history of changes, resolving conflict andothers. Main characteristic of the Git CVS is strong support for a dis-tributed development. For instance, each developer uses its local repo-sitory for work, thus changes are committed to a local repository atfirst, then pushed to a remote repository. In order to provide a web-based service for Git repositories, the Github3 was created. It allowsuser to manage its repositories via web interface.

5.3.4 Kibana

Elasticsearch stores large volumes of data that a user needs to workwith. Kibana allows a user to search, view and analyze data withinthe Elasticsearch. It represents a simple web interface for browsingdata in real time. Moreover, Kibana provides advanced data analysisand visualization using various charts, tables and maps [14, Intro-duction]. Note that installing and running Kibana does not requireany additional implementation on the client side, thus using Kibanais very simple and intuitive.

5.3.5 Bytecode manipulation tools

The non-functional requirement of this work is to use an approachwhich allows a user to add log events to Wildfly and does not changeits source code. The Wildfly source code is written in the Java lan-guage and compiled into Bytecode. Thus, new log events must beadded only by modifying the Wildfly bytecode, whereas the currentsource code is left untouched. To fulfill such requirement, the byte-code manipulation must be employed. There are several bytecodemanipulation tools depending on a level of abstraction they use. Thefollowing picture shows selected tools and the level they operate on:

3. https://github.com/

33

5. Analysis and Design

Figure 5.2: Bytecode manipulation tools

Choosing a suitable bytecode manipulation tool is a critical deci-sion as all additional log events are implemented using the selectedtool. Thus, replacing one tool with another requires the source codeto be rewritten, which is not trivial operation in case of large numberof log events. Selection of a bytecode library does not depend onlyon the architecture of the library itself, it also requires analysis of thelibrary used against an application server which can be implementedin various ways. For instance, Wildfly application server uses a modu-lar classloading system 2.2 that can cause classloading problems withspecific bytecode library. Another requirement on the selected byte-code library is reusability of implemented log events across differentreleases of Wildfly. For this reason, the bytecode library should pro-vide bytecode analysis which is able to target specific parts withinbytecode based on class names, method names, method invocations,variable assignments, constructor invocations and others.

The following bytecode manipulation tools were chosen for anal-ysis and testing: Byteman and AspectJ4. There is a significant distinc-tion between Byteman and AspectJ as they use different approach forbytecode manipulation. Byteman is a low level library which requireswork at bytecode level, whereas AspectJ uses Java classes to achievemanipulation with bytecode. After an appropriate analysis based onthe documentation and testing of selected tools, Byteman is used bythe following reasons:

4. https://eclipse.org/aspectj/

34

5. Analysis and Design

• Byteman uses a simple and powerful mechanism (see 3.2.1) toperform bytecode manipulation

• AspectJ requires additional configuration using .xml files fordefining and working with aspects [16, Configuration]

• Byteman Rule Language 3.2 provides a comprehensive byte-code analysis 3.2.2 to identify target location within bytecode

• in contrast to Byteman, AspectJ does not allow to manipulatewith Java language classes defined in the java.lang package [16,Special cases]

• Byteman provides a set of built-in operations 3.2.6 which can beused for debugging, monitoring and thread coordinating pur-poses, thus a user does not have to implement such operationsfor specific requirements

• both Byteman and Wildfly are part of the JBoss Community, thisfact can simplify co-operation between developers in case of ad-ditional requirements

35

6 Implementation

This chapter describes the implementation of extended Wildfly log-ging capabilities in detail. The logging extension is split into two Mav-en projects:

• elasticsearch-wildfly-log: extension for Elasticsearch

• byteman-wildfly-log: addition of new log events using Byteman

Both Maven projects are completely independent and can be usedseparately.

6.1 Elasticsearch Wildfly Log

Main functionality of the Elasticsearch Wildfly Log project is to storelog events to Elasticsearch. This project is deployed as a static modulein Wildfly. Thus, the project is built as a directory, which correspondsto the JBoss Module hierarchical directory structure depicted in fig-ure 2.2. The output directory has the following structure:

org|--elasticsearch| |--main| |--elasticsearch -1.6.0. jar| |--lucene -core -4.10.4. jar| |-- ... rest of Lucene jars| |-- module.xml|--wildfly

|--elasticsearch|--log

|--main|--elasticsearch -wildfly -log -0.0.1 -

SNAPSHOT.jar|--module.xml

Listing 6.1: Elasticsearch Wildfly Log module directory structure

36

6. Implementation

Note that there are two modules in the output directory as the org.wil-dfly.elasticsearch.log module requires the Elasticsearch jars which areincluded in the separate module located in the org.elasticsearch subdi-rectory.

The storing process of log events includes several steps which areimplemented in the ElasticsearchHandler class. At first, the handlermust establish a connection with an Elasticsearch cluster. To createsuch connection, a user must specify valid cluster name, transport ad-dress and port. If correct configuration is provided, the handler im-plementation uses the Transport Client 4.3.2 to connect to a runningcluster.

The next step in the storing process is to serialize a log event tothe JSON format, which is used for data stored in Elasticsearch. TheJSON format of the log event is designed to provide a comprehensivedata structure in order to support search operations in Elasticsearch.

{"timestamp":"timestamp","jboss.bind.address":"server address","jboss.server.name":"server name","level":"log level","loggerName":"loger name","message":"log message","thrown":{

"name":"throwable name","message":"throwable message"

}}

Listing 6.2: JSON format of a log event

Note that the jboss.bind.address and jboss.server.name values are ob-tained from Wildfly system properties. Information about the servername and address is important in case of multiple server instances(e.g. cluster or cloud installation).

When a log event is successfully converted to the JSON format,transport client sends the log event to an Elasticsearch cluster whichinserts received data into specific index and type. The index and type

37

6. Implementation

are defined by a user when configuring the Elasticsearch handler inWildfly logging subsystem (see 2.3).

6.2 Byteman Wildfly Log

This project is comprised of tools which provide bytecode manipu-lation in order to inject new log events into the Wildfly bytecode.There are two main elements which forms the logging mechanism:LogHelper class and Byteman rules.

6.2.1 Log Helper class

The LogHelper class represents a Byteman rule helper described in3.2.7. This class uses java.util.logging.LogManager to obtain a specificlogger instance according to provided logger name. The logger in-stance is used to create a log event with the following parameters:level, message and Throwable associated with the log event. The fullpackage qualified logger name and parameters are passed by a byte-man rule which uses the log helper class. The LogHelper class containstwo public methods:

public void log(String loggerName , String level ,String message)

public void log(String loggerName , String level ,String message , Throwable thrown)

Listing 6.3: LogHelper methods for creating log events.

The methods above are responsible for creating log events which canhave an exception attached to them.

6.2.2 Byteman rules

In order to employ the LogHelper class described in previous section,the helper class must be used within a Byteman rule. The BytemanWildfly Log project consists of five rule script files. Each rule scriptfile defines a set of Byteman rules which add new log events to a

38

6. Implementation

particular part of Wildfly. The following rule script files cover se-lected parts of Wildfly specified in the requirements section (5.1):servelts.btm, rest.btm, ws.btm, ejb.btm, jms.btm.

The servlet.btm rule file contains log events for JSP configurationand the entire chain of servlet handlers including security context,thread context, HTTP metadata and others.

Additional log events for Wildfly REST implementation are de-fined in the rest.btm rule file. The log events provide more detailedinformation about REST procedures such as initializing resources,parsing parameters, validation of parameters and return value, in-vocation of resource method, building response.

The ws.btm rule file provides logging extension for Wildfly WebServices. The extension adds new log events to request handler call,manipulation with bus and endpoint metadata, initialization of WSbean, invoking WS bean and closing resources.

Extension for the Wildfly EJB subsystem contains the largest amo-unt of new log events because of the complexity of the EJB imple-mentation. The rules in the ejb.btm rule script file provide additionalmonitoring of a stateful/stateless EJB and its entire interceptor chain,for example: method invocation on a proxy instance, security permis-sions, manipulation with interceptor context (thread stack, contextmetadata etc.), transaction processing, concurrency, component in-stance lifecycle, EJB request scope processing, stateful synchroniza-tion process, JSR2991, creating interceptor instance, invoking inter-cepted method and many others.

The jms.btm rule file focuses on the JMS subsystem. The rules addnew log events to the JMS channel workflow, packet handling, contextand session operations, processing queue and executing queue query,transactions, sending message and message expiration.

The following example from servlets.btm rule script illustrates im-plementation of a new log event injected into the Wildfly bytecode:

1. JSR 299: Contexts and Dependency Injection - The purpose of this specifica-tion is to unify the JSF managed bean component model with the EJB componentmodel, resulting in a significantly simplified programming model for web-basedapplications [17].

39

6. Implementation

HELPER org.jboss.byteman.koubsky.LogHelperRULE logServletHandler.handleRequestCLASS io.undertow.servlet.handlers.

ServletHandlerMETHOD handleRequest(HttpServerExchange)AT ENTRYIF trueDO log($CLASS , "DEBUG", "(Method handleRequest)

Request info: " +" URL = " + $1. getRequestURL () + ", protocol = "

+ $1. getProtocol () +", method = " + $1. getRequestMethod () + ", query

string: " + $1. getQueryString ())ENDRULE

Listing 6.4: An example of the rule implementation for a new logevent

The rule above adds a new log event to the io.undertow.servlet.handlers.-ServletHandler class when entering the handleRequest(HttpServerExcha-nge) method. The message string is built using the $ expression (3.2.4)which provides access to runtime values of the trigger method.

40

7 Conclusion

The purpose of this work is to extend Wildly Application Server log-ging capabilities. The target version is Wildfly 8, however, the imple-mentation is designed to be reusable in later releases of Wildfly.

The logging extension consists of two independent projects hostedon Github: Elasticsearch Wildfly Log1 and Byteman Wildfly Log2.

The Elasticsearch Wildfly Log is able to store log events to Elas-ticsearch from multiple server instances (e.g. cluster or cloud instal-lation). The data model of the log event stored in Elasticsearch is de-signed to be easily analyzed by Kibana tool. Although the implemen-tation was not tested under heavy load, Wildfly and Elasticsearchprovide solutions for processing large amount of log events. Whenthe project was finished, an article3 about similar idea was written byJames R. Perkins. Perkins introduced the ELK (Elasticsearch, Logstash,Kibana) stack and its use to store log events from Wildfly to Elas-ticsearch. However, the ELK stack is standalone solution apart fromWildfly, thus specific requirements on the server side cannot be im-plemented.

The Byteman Wildfly Log provides injecting new log events intoWildlfy bytecode. Because of the bytecode injection, there is no needto change the source code of the server, thus the new log events canbe applied to different versions of Wildfly if there are no significantchanges in the source code. In summary, 226 log events was added toselected parts of Wildfly: Web Container (Servlets, JSPs), REST end-points, WS endpoints, EJB invocations and JMS communication.

Adding log events using Byteman may seem to be inappropriateas log events can be added directly to source code. Advantage of thisapproach is a unified process of adding log events in a broader con-text than only in a context of a certain method within a class. Thus,it is possible to manipulate with log events in a context of a compo-nent or a whole system. For example, a thread id is attached to all logevents added using Byteman. In addition, the Elasticsearch extensionattaches server address and server name to every log event within

1. https://github.com/RadekKoubsky/elasticsearch-wildfly-log2. https://github.com/RadekKoubsky/byteman-wildfly-log3. http://wildfly.org/news/2015/07/25/Wildfly-And-ELK/

41

7. Conclusion

Wildfly, thus log events from specific server can be easily identifiedin case of multiple server instances.

Two bugs, BYTEMAN-2984 and BYTEMAN-2995, were found dur-ing the implementation of Byteman rules. Thanks to Andrew Dinnand the author of this work, both bugs were fixed and verified in Byte-man 3.0.2 release.

One critical issue emerged from Byteman and Wildfly coopera-tion. When a Byteman rule passes a class from JBoss Modules system2.2 to the Log Helper class 6.2.1, the java.lang.ClassNotFoundExceptionis thrown because the Log Helper class is loaded by different class-loader. After a discussion with Andrew Dinn, an experimental solu-tion of the classloading problem was implemented and is availablein Byteman 3.0.3 release in form of JBoss Modules plugin6. How-ever, Byteman 3.0.2 was released 6 months after the original imple-mentation of this work. Thus, Byteman rules must use the $ expres-sion to build log messages and pass them to the Log Helper class asjava.lang.String.

Future work should focus on performance tests in order to provideresults for further analysis of the logging extension. In addition, theoriginal implementation of Byteman rules can be rewritten using theJBoss Modules plugin which solves the classloading problem.

4. https://issues.jboss.org/browse/BYTEMAN-2985. https://issues.jboss.org/browse/BYTEMAN-2996. http://bytemanblog.blogspot.cz/2015/11/byteman-303-release-trials-module.html

42

Bibliography

[1] S. Gupta, Logging in Java with the JDK 1.4 Logging API and Apachelog4j. APress, 2003, ISBN: 978-1590590997.

[2] JBoss. (2015, Mar.) WildFly 8 Documentation. [On-line]. Available: https://docs.jboss.org/author/display/WFLY8/Documentation

[3] F. Marchioni, JBoss AS 7 Development: develop, deploy, and secureJava applications on the new release of this robust, open source applica-tion server. Packt Publishing Ltd., 2013, ISBN: 978-1-78216-134-9.

[4] C. Ritchie, WildFly Configuration, Deployment, and AdministrationSecond Edition. 35 Livery Street, Birmingham B3 2PB, UK: PacktPublishing Ltd., 2014, ISBN: 978-1783286232.

[5] JBoss. (2011, Jul.) JBoss Modules Documentation.[Online]. Available: https://docs.jboss.org/author/display/MODULES/Home

[6] ——. (2015, May) Byteman Programmer’s Guide. [On-line]. Available: http://downloads.jboss.org/byteman/3.0.0/ProgrammersGuide.pdf

[7] J. J. Alferes, F. Banti, and A. Brogi. (2015) An Event-Condition-Action Logic Programming Language. [Online]. Available:http://www.di.unipi.it/~brogi/papers/JELIA06.pdf

[8] S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Ap-proach. Prentice-Hall, Inc, 1995, ISBN: 0-13-103805-2.

[9] W. Binder, J. Hulaas, and P. Moret. (2015) ReengineeringStandard Java Runtime Systems through Dynamic BytecodeInstrumentation. [Online]. Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4362901

[10] C. Gormley and Z. Tong, Elasticsearch: The Definitive Guide.O’Reilly Media, Inc, 2015, ISBN: 978-1-4493-5854-9.

43

BIBLIOGRAPHY

[11] R. T. Fielding and R. N. Taylor. (2000) Principled Designof the ModernWeb Architecture. [Online]. Available: https://www.ics.uci.edu/~fielding/pubs/webarch_icse2000.pdf

[12] Elasticsearch. (2015) Elasticsearch Reference 2.0. [Online].Available: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

[13] ——. (2015) Java API 2.0. [Online]. Avail-able: https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.0/index.html

[14] ——. (2015) Kibana. [Online]. Available: https://www.elastic.co/guide/en/kibana/4.3/index.html

[15] The Apache Software Foundation. (2015) Apache Maven.[Online]. Available: https://maven.apache.org/what-is-maven.html

[16] AspectJ. (2015) The AspectJ Development Environment Guide.[Online]. Available: https://eclipse.org/aspectj/doc/released/devguide/index.html

[17] Java Community Process. (2015) JSR 299: Contexts and Depen-dency Injection for the Java EE platform. [Online]. Available:https://jcp.org/en/jsr/detail?id=299

44