Testability Transformation - Program Transformation to Improve Testability

AK Softwaretechnologie 2

Testability Transformation: ProgramTransformation to Improve Testability - An

Overview of Recent ResultsMark Harman, Andre Baresel, David Binkley, Robert Hierons,

Lin Hu, Bogdan Korel, Phil McMinn, Marc Roper

Christian Wutte

June 8, 2011

Christian Wutte June 8, 2011 1/31

Overview

Introduction

Testability Transformation

Test Data Generation Techniques

TeTra for Evolutionary Testing: Flag Problem

TeTra for Evolutionary Testing: Nested Predicates

TeTra for Chaining Method

Open? questions

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Introduction

Testing is based on structurally defined test adequacy criteria.Properties like e.g. flag variables, side effects or unstructuredcontrol flow hinder automated generation of test data to satisfycriteria.How can those properties be transformed without changingstructure and therefore structurally defined test adequacy criteria?

AnswerAllow transformation to co-transform criterion.

Criteria examples

Set of nodes, branches, paths, etc.

Criteria examples

DefinitionLet P be a set of programs and C a set of test adequacy criteria. ATesting- Oriented Transformation is a partial function in (P x C)� (P x C). (In general, a program transformation is a partialfunction in P � P).A Testing-Oriented Transformation τ is a TestabilityTransformation iff for all programs p and criteria c, τ(p, c) = (p´,c´) implies that for all test sets T, T is adequate for p according toc if T is adequate for p´according to c´.

Test Data Generation Techniques

Techniques

Evolutionary testingChaining method

Evolutionary testing

Based on evolutionary algorithm

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is usedExample guidance: a > bCase where a == b more likely chosen then a < b

Fitness function

Example function: test branch, so a condition a == b should betrue

� fitness function abs(a-b) is usedExample guidance: a > bCase where a == b more likely chosen then a < b

Fitness function

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is used

Example guidance: a > bCase where a == b more likely chosen then a < b

Fitness function

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is usedExample guidance: a > b

Case where a == b more likely chosen then a < b

Fitness function

Chaining Method

Relies on data dependency analysis which is used to acquired asequence of statements (event) which affect a target statement.This sequence will be exectued with the variable method

I Exploratory phase (based on local search), small steps

I Pattern phase, ever-larger steps

I If target not found: next sequence.

Chaining Method

DefinitionA flag variable is any boolean variable used as a predicate.Few inputs (those in a Set S) map to one of the two possiblevalues, all other inputs (not in Set S) being mapped to the other.

I Hard to find a value from S.

I No guidance: fitness function has only two results (maximumvalue for an input from S or minimum value for an input notin S).

Flag problem by example (with loop)

Similar k-valued enum

I Flag variable corresponse to 2-valued enum.

I For larger k testability increases and so the landscape acquiresmore guidance.

Similar k-valued enum

I Flag variable corresponse to 2-valued enum.

I For larger k testability increases and so the landscape acquiresmore guidance.

Loop-assign flag removal algorithm (coarse) by example

Loop-assign removal algorithm (fine-grained) by example

Flag problem results

Flag problem results: no transformation

Flag problem results: coarse transformation

Flag problem results: fine transformation

Key problems

I Input constraints are revealed after each individual predicate issatisfied

I Search guidance is treated locally at each predicate

Key problems

Example

SolutionIn essence: all conditions leading to the target are evaluated at thesame time.Made possible by removing decision statements on which thetarget is control.

Solution

Solution issuesIssues if removed conditions prevent run-time errors (e.g. check ifpointer null) � test object evaluation in own sandbox

Results

I 40 programs showed prevalence of nested predicates (onaverage 3 nested predicate pairs per 100 loc)

I Half the test data evaluations for example program, foranother program without transformation no, withtransformation every time test data was found.

Speculative TeTra application

I Transformation in such way, that it is known that wrong resultcan result.

I But, as a speculatively approach, it can give test data whereother techniques fail.

I More expensive

Example

Data dependency transformation

I Data dependency subgraph creation

I Based on the graph a transformed program is generated

I Test data generation for the transformed program

I With the test data search in original program is guided

Data dependency subgraph

Transformed program

I Each statement of graph is included as switch-case statement.

I Additional input arrays S[] und R[] (S ... statements, R ...repeats)

Transformed program

Open? questions

I We have seen flag variables, nested predicates andunstructured control flow. Open issues: internal state,continous input and loop problem for symbolic execution.

I Semantic foundations for semantic correctness?

I Testing on model level, TeTra for formal specifications?

I Maybe other ”Testing Oriented Transformations” besideTestability transformation are useful?

Thank you for your attention.

Testability Transformation - Program Transformation to Improve Testability

Documents

Transcript of Testability Transformation - Program Transformation to Improve Testability