Testability Transformation - Program Transformation to Improve Testability

72
AK Softwaretechnologie 2 Testability Transformation: Program Transformation to Improve Testability - An Overview of Recent Results Mark Harman, Andre Baresel, David Binkley, Robert Hierons, Lin Hu, Bogdan Korel, Phil McMinn, Marc Roper Christian Wutte June 8, 2011 Christian Wutte June 8, 2011 1/31

Transcript of Testability Transformation - Program Transformation to Improve Testability

AK Softwaretechnologie 2

Testability Transformation: ProgramTransformation to Improve Testability - An

Overview of Recent ResultsMark Harman, Andre Baresel, David Binkley, Robert Hierons,

Lin Hu, Bogdan Korel, Phil McMinn, Marc Roper

Christian Wutte

June 8, 2011

Christian Wutte June 8, 2011 1/31

AK Softwaretechnologie 2

Overview

Introduction

Testability Transformation

Test Data Generation Techniques

TeTra for Evolutionary Testing: Flag Problem

TeTra for Evolutionary Testing: Nested Predicates

TeTra for Chaining Method

Open? questions

Christian Wutte June 8, 2011 2/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Introduction

Testability transformation (TeTra): program transformation toimprove performance of test data generation technique

Differences to traditional program transformation:

I transformed program is a ’means to an end’, important is theresulting test data

I no psychological barrier (code changes by machine)

I no functional equivalence needed, only equivalence regardingthe test criteria

I more flexibility in transformation choice

I correctness less important (if incorrect only test data may bewrong, easy detectable / functional equivalence undecideable)

Christian Wutte June 8, 2011 3/31

AK Softwaretechnologie 2

Testability Transformation

Testing is based on structurally defined test adequacy criteria.Properties like e.g. flag variables, side effects or unstructuredcontrol flow hinder automated generation of test data to satisfycriteria.How can those properties be transformed without changingstructure and therefore structurally defined test adequacy criteria?

AnswerAllow transformation to co-transform criterion.

Criteria examples

Set of nodes, branches, paths, etc.

Christian Wutte June 8, 2011 4/31

AK Softwaretechnologie 2

Testability Transformation

Testing is based on structurally defined test adequacy criteria.Properties like e.g. flag variables, side effects or unstructuredcontrol flow hinder automated generation of test data to satisfycriteria.How can those properties be transformed without changingstructure and therefore structurally defined test adequacy criteria?

AnswerAllow transformation to co-transform criterion.

Criteria examples

Set of nodes, branches, paths, etc.

Christian Wutte June 8, 2011 4/31

AK Softwaretechnologie 2

Testability Transformation

Testing is based on structurally defined test adequacy criteria.Properties like e.g. flag variables, side effects or unstructuredcontrol flow hinder automated generation of test data to satisfycriteria.How can those properties be transformed without changingstructure and therefore structurally defined test adequacy criteria?

AnswerAllow transformation to co-transform criterion.

Criteria examples

Set of nodes, branches, paths, etc.

Christian Wutte June 8, 2011 4/31

AK Softwaretechnologie 2

Testability Transformation

DefinitionLet P be a set of programs and C a set of test adequacy criteria. ATesting- Oriented Transformation is a partial function in (P x C)� (P x C). (In general, a program transformation is a partialfunction in P � P).A Testing-Oriented Transformation τ is a TestabilityTransformation iff for all programs p and criteria c, τ(p, c) = (p´,c´) implies that for all test sets T, T is adequate for p according toc if T is adequate for p´according to c´.

Christian Wutte June 8, 2011 5/31

AK Softwaretechnologie 2

Test Data Generation Techniques

Techniques

Evolutionary testingChaining method

Christian Wutte June 8, 2011 6/31

AK Softwaretechnologie 2

Evolutionary testing

Based on evolutionary algorithm

Christian Wutte June 8, 2011 7/31

AK Softwaretechnologie 2

Evolutionary testing

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is usedExample guidance: a > bCase where a == b more likely chosen then a < b

Christian Wutte June 8, 2011 8/31

AK Softwaretechnologie 2

Evolutionary testing

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is usedExample guidance: a > bCase where a == b more likely chosen then a < b

Christian Wutte June 8, 2011 8/31

AK Softwaretechnologie 2

Evolutionary testing

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue

� fitness function abs(a-b) is usedExample guidance: a > bCase where a == b more likely chosen then a < b

Christian Wutte June 8, 2011 8/31

AK Softwaretechnologie 2

Evolutionary testing

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is used

Example guidance: a > bCase where a == b more likely chosen then a < b

Christian Wutte June 8, 2011 8/31

AK Softwaretechnologie 2

Evolutionary testing

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is usedExample guidance: a > b

Case where a == b more likely chosen then a < b

Christian Wutte June 8, 2011 8/31

AK Softwaretechnologie 2

Evolutionary testing

Fitness function

I Depends on the test aim (temporal behavior, safety,robustness, etc.)

I Here: structural criteria, so fitness function defined in termsof program’s predicates

I Measures how close a candidate is to the target (path,branch) and which direction is taken next (guidance)

Example function: test branch, so a condition a == b should betrue� fitness function abs(a-b) is usedExample guidance: a > bCase where a == b more likely chosen then a < b

Christian Wutte June 8, 2011 8/31

AK Softwaretechnologie 2

Chaining Method

Relies on data dependency analysis which is used to acquired asequence of statements (event) which affect a target statement.This sequence will be exectued with the variable method

I Exploratory phase (based on local search), small steps

I Pattern phase, ever-larger steps

I If target not found: next sequence.

Christian Wutte June 8, 2011 9/31

AK Softwaretechnologie 2

Chaining Method

Relies on data dependency analysis which is used to acquired asequence of statements (event) which affect a target statement.This sequence will be exectued with the variable method

I Exploratory phase (based on local search), small steps

I Pattern phase, ever-larger steps

I If target not found: next sequence.

Christian Wutte June 8, 2011 9/31

AK Softwaretechnologie 2

Chaining Method

Relies on data dependency analysis which is used to acquired asequence of statements (event) which affect a target statement.This sequence will be exectued with the variable method

I Exploratory phase (based on local search), small steps

I Pattern phase, ever-larger steps

I If target not found: next sequence.

Christian Wutte June 8, 2011 9/31

AK Softwaretechnologie 2

Chaining Method

Relies on data dependency analysis which is used to acquired asequence of statements (event) which affect a target statement.This sequence will be exectued with the variable method

I Exploratory phase (based on local search), small steps

I Pattern phase, ever-larger steps

I If target not found: next sequence.

Christian Wutte June 8, 2011 9/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

DefinitionA flag variable is any boolean variable used as a predicate.Few inputs (those in a Set S) map to one of the two possiblevalues, all other inputs (not in Set S) being mapped to the other.

I Hard to find a value from S.

I No guidance: fitness function has only two results (maximumvalue for an input from S or minimum value for an input notin S).

Christian Wutte June 8, 2011 10/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

DefinitionA flag variable is any boolean variable used as a predicate.Few inputs (those in a Set S) map to one of the two possiblevalues, all other inputs (not in Set S) being mapped to the other.

I Hard to find a value from S.

I No guidance: fitness function has only two results (maximumvalue for an input from S or minimum value for an input notin S).

Christian Wutte June 8, 2011 10/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

DefinitionA flag variable is any boolean variable used as a predicate.Few inputs (those in a Set S) map to one of the two possiblevalues, all other inputs (not in Set S) being mapped to the other.

I Hard to find a value from S.

I No guidance: fitness function has only two results (maximumvalue for an input from S or minimum value for an input notin S).

Christian Wutte June 8, 2011 10/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

DefinitionA flag variable is any boolean variable used as a predicate.Few inputs (those in a Set S) map to one of the two possiblevalues, all other inputs (not in Set S) being mapped to the other.

I Hard to find a value from S.

I No guidance: fitness function has only two results (maximumvalue for an input from S or minimum value for an input notin S).

Christian Wutte June 8, 2011 10/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem by example (with loop)

Christian Wutte June 8, 2011 11/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem by example (with loop)

Christian Wutte June 8, 2011 11/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem by example (with loop)

Christian Wutte June 8, 2011 11/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Similar k-valued enum

I Flag variable corresponse to 2-valued enum.

I For larger k testability increases and so the landscape acquiresmore guidance.

Christian Wutte June 8, 2011 12/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Similar k-valued enum

I Flag variable corresponse to 2-valued enum.

I For larger k testability increases and so the landscape acquiresmore guidance.

Christian Wutte June 8, 2011 12/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Loop-assign flag removal algorithm (coarse) by example

Christian Wutte June 8, 2011 13/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Loop-assign flag removal algorithm (coarse) by example

Christian Wutte June 8, 2011 13/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Loop-assign flag removal algorithm (coarse) by example

Christian Wutte June 8, 2011 13/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Loop-assign removal algorithm (fine-grained) by example

Christian Wutte June 8, 2011 14/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Loop-assign removal algorithm (fine-grained) by example

Christian Wutte June 8, 2011 14/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Loop-assign removal algorithm (fine-grained) by example

Christian Wutte June 8, 2011 14/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem results

Christian Wutte June 8, 2011 15/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem results

Christian Wutte June 8, 2011 15/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem results: no transformation

Christian Wutte June 8, 2011 16/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem results: coarse transformation

Christian Wutte June 8, 2011 17/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Flag Problem

Flag problem results: fine transformation

Christian Wutte June 8, 2011 18/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Key problems

I Input constraints are revealed after each individual predicate issatisfied

I Search guidance is treated locally at each predicate

Christian Wutte June 8, 2011 19/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Key problems

I Input constraints are revealed after each individual predicate issatisfied

I Search guidance is treated locally at each predicate

Christian Wutte June 8, 2011 19/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Key problems

I Input constraints are revealed after each individual predicate issatisfied

I Search guidance is treated locally at each predicate

Christian Wutte June 8, 2011 19/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Example

Christian Wutte June 8, 2011 20/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Example

Christian Wutte June 8, 2011 20/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

SolutionIn essence: all conditions leading to the target are evaluated at thesame time.Made possible by removing decision statements on which thetarget is control.

Christian Wutte June 8, 2011 21/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Solution

Christian Wutte June 8, 2011 22/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Solution

Christian Wutte June 8, 2011 22/31

AK Softwaretechnologie 2

TeTra for Evolutionary Testing: Nested Predicates

Solution issuesIssues if removed conditions prevent run-time errors (e.g. check ifpointer null) � test object evaluation in own sandbox

Results

I 40 programs showed prevalence of nested predicates (onaverage 3 nested predicate pairs per 100 loc)

I Half the test data evaluations for example program, foranother program without transformation no, withtransformation every time test data was found.

Christian Wutte June 8, 2011 23/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Speculative TeTra application

I Transformation in such way, that it is known that wrong resultcan result.

I But, as a speculatively approach, it can give test data whereother techniques fail.

I More expensive

Christian Wutte June 8, 2011 24/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Speculative TeTra application

I Transformation in such way, that it is known that wrong resultcan result.

I But, as a speculatively approach, it can give test data whereother techniques fail.

I More expensive

Christian Wutte June 8, 2011 24/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Speculative TeTra application

I Transformation in such way, that it is known that wrong resultcan result.

I But, as a speculatively approach, it can give test data whereother techniques fail.

I More expensive

Christian Wutte June 8, 2011 24/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Speculative TeTra application

I Transformation in such way, that it is known that wrong resultcan result.

I But, as a speculatively approach, it can give test data whereother techniques fail.

I More expensive

Christian Wutte June 8, 2011 24/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Example

Christian Wutte June 8, 2011 25/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Data dependency transformation

I Data dependency subgraph creation

I Based on the graph a transformed program is generated

I Test data generation for the transformed program

I With the test data search in original program is guided

Christian Wutte June 8, 2011 26/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Data dependency transformation

I Data dependency subgraph creation

I Based on the graph a transformed program is generated

I Test data generation for the transformed program

I With the test data search in original program is guided

Christian Wutte June 8, 2011 26/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Data dependency transformation

I Data dependency subgraph creation

I Based on the graph a transformed program is generated

I Test data generation for the transformed program

I With the test data search in original program is guided

Christian Wutte June 8, 2011 26/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Data dependency transformation

I Data dependency subgraph creation

I Based on the graph a transformed program is generated

I Test data generation for the transformed program

I With the test data search in original program is guided

Christian Wutte June 8, 2011 26/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Data dependency transformation

I Data dependency subgraph creation

I Based on the graph a transformed program is generated

I Test data generation for the transformed program

I With the test data search in original program is guided

Christian Wutte June 8, 2011 26/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Data dependency subgraph

Christian Wutte June 8, 2011 27/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Transformed program

I Each statement of graph is included as switch-case statement.

I Additional input arrays S[] und R[] (S ... statements, R ...repeats)

Christian Wutte June 8, 2011 28/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Transformed program

I Each statement of graph is included as switch-case statement.

I Additional input arrays S[] und R[] (S ... statements, R ...repeats)

Christian Wutte June 8, 2011 28/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Transformed program

I Each statement of graph is included as switch-case statement.

I Additional input arrays S[] und R[] (S ... statements, R ...repeats)

Christian Wutte June 8, 2011 28/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Transformed program

Christian Wutte June 8, 2011 29/31

AK Softwaretechnologie 2

TeTra for Chaining Method

Transformed program

Christian Wutte June 8, 2011 29/31

AK Softwaretechnologie 2

Open? questions

I We have seen flag variables, nested predicates andunstructured control flow. Open issues: internal state,continous input and loop problem for symbolic execution.

I Semantic foundations for semantic correctness?

I Testing on model level, TeTra for formal specifications?

I Maybe other ”Testing Oriented Transformations” besideTestability transformation are useful?

Christian Wutte June 8, 2011 30/31

AK Softwaretechnologie 2

Thank you for your attention.

Christian Wutte June 8, 2011 31/31