Testing TAROT Summer School Lustre/SCADE programs 2009 -...
Transcript of Testing TAROT Summer School Lustre/SCADE programs 2009 -...
1
(TAROT 2009)
Testing Lustre/SCADE programs
Lydie du Bousquet, Ioannis Parissis
TAROT Summer School 2009 - July 6-10
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
2
high level
of criticality
control/command
programs
Synchronous / safety-critical software
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
3
Reactive systems
Environment system
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
4
Synchronous program
Synchronous hypothesis : time reaction is null
Environment Program
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
5
• Esterel, Signal, Lustre
• SCADE (Safety-Critical Applications Development Environment)– Based on Lustre
Synchronous languages
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
6
SCADE / Lustre overview
• Graphical/textual
• Simple constructions
• Flows and clocks
node never (E: bool) returns (S : bool);
let
S = not(E) -> (not(E) and pre(S));
tel;
E
S
pre
L1
L4
L2L3
E = (E1, E2, E3, … ) S = (S1, S2, S3, … )
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
7
Example of Lustre program
node never (E: bool) returns (S : bool);
let
S = not(E) -> (not(E) and pre(S));
tel;
clockinput
output
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
8
Scade V6
• Automaton• Data-flow
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
9
Using Scade in Avionics : development approach
R1R2
High-level requirements
C-code
C files
Scade files
Low-level requirements
E
S
pre
L1L4
L2
L3
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
10
Using Scade in Avionics : validation problems
R1R2
Covered?
C files
Scade files
Correct?E
S
pre
L1L4
L2
L3
Correct?
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
11
Using Scade in Avionics : validation problems
R1R2
Covered?
Scade files
Correct?E
S
pre
L1L4
L2
L3
How to produce tests efficiently ?
Tests produced • Manually• From requirement • MC/DC coverage
Tracability issuesNo dead code
DO – 178 B
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
12
The SIESTA project (jan 2008 – jan 2011) Automation of testing in SCADE based development
Toulouse Pau
Paris
GrenobleValence
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
13
SIESTA project: Automation of testing in SCADE based development
Informal requirements
Test objectives
Test specification
Test generation
Test execution
Evaluation
R1R2
C filesE
Scade
Evaluation of research tools for generation
Evaluation of research tools
for coverage / testability
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
(TAROT 2009)
Objective : automation of the testing process for Scade/Lustre programs
What are the issues ?
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
15
Selection of data for Scade/Lustre : issues
Environment Program
(E1, E2, E3, … )
(S1, S2, S3, … )
Test data : sequences of inputs/outputstemporal dependencies (pre, ->, multi-clocks…)
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
16
Stopping criteria : issues with Lustre
Data-flow and equational: All equations evaluated No graph control
node never (E: bool) returns (S : bool);
let
S = not(E) -> (not(E) and pre(S));
tel;
E
S
pre
L1
L4
L2L3
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
17
Decision and tracability issues
R1R2
Scade files
E
S
pre
L1L4
L2
L3Informal vs formal
IntroductionGenerationStoppingEvaluation
SynchroneScade/LustreSiestaIssues
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
18
Outline
• Lustre / SCADE
• Test generation
• Stopping criteria
• Evaluation
IntroductionGenerationStoppingEvaluation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
19
How to produce test data ?
• Specification / requirement– using the description of its environment– using the specification of the expected behaviours
(test objectives)
• Code
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
20
Outline
• Lustre / SCADE
• Test generation– Using the description of the environment
– Using the code
• Stopping criteria
• Evaluation
IntroductionGenerationStoppingEvaluation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
21
Environment
Testing using the environment description
Simulation ofthe environment
System
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
22
Lutess : a tool for testing Lustre programs
input input generatorgenerator
properties
Oracle
Environmentdescription(Lustre)
Programunder test
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
23
Lutess : a tool for testing Lustre programs (2)
input input generatorgenerator
properties
Programunder test
Oracle
eeee
ssss
e’e’e’e’
ssee
vvLutess V2 : based
on constrain programming)
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
24
A (very) simple air conditionner controller
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
25
Lustre implementation of a AC controller
node Program_AC(Button : bool ; Tamb, Tuser : int)returns (OnOff : bool ; Tout : int) ;let OnOff = Button -> ((pre OnOff) and not Button) or (not(pre OnOff) and Button) ; Tout = Tuser + (Tuser - Tamb) / 3 ;tel
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
26
Behavior of the AC controller
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
27
Using Lutess for testing the AC
input generatorinput generator
Program_ac.exe
Program under test
testnode TestAC(…) returns (…) ;let -- env. specificationtel ;
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
28Using Lutess for testing the AC Physical invariants : domain definition
« There are constraints on the temperature range »
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
29Using Lutess for testing the AC Physical invariants : domain definition
testnode TestAC(OnOff : bool ; Tout : int) returns (Button : bool ; Tamb, Tuser : int) ;let environment(
(Tamb >= -20) and (Tamb <= 60) and (Tuser >= 10) and (Tuser <= 40)) ;tel ;
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
30
IntroductionGenerationStoppingEvaluation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
31
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
32
Using Lutess for testing the AC Physical invariants : system dynamics
« No big variations of the ambient temperature »
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
33
Using Lutess for testing the AC Physical invariants : system dynamics
testnode TestAC(OnOff : bool ; Tout : int) returns (Button : bool ; Tamb, Tuser : int) ;var dT : int ;let environment(
(Tamb >= -20) and (Tamb <= 60) and (Tuser >= 10) and (Tuser <= 40) and
(dT >= -1) and (dT <= 1) ) ; dT = 0 -> Tamb - pre Tamb ;tel ;
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
34
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
35
Helping the random generation
input generatorinput generator
Program_ac.exe
Program under test
testnode TestAC(…) returns (…) ;let -- specification -- scenarios -- operational profilestel ;
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
36
Using Lutess for testing the AC Guiding the generation with scenarios
« If the AC is off, then the user will turn it on »
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
37
Using Lutess for testing the AC Guiding the generation with scenarios
testnode TestAC (…) returns (…) ;var dT : int ;let environment( true->(implies(not(pre(OnOff)), Button) and implies(pre(OnOff), not Button))); … dT = 0 -> Tamb - pre Tamb ;tel ;
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
38
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
39
Helping the random generation
input generatorinput generator
Program_ac.exe
Program under test
testnode TestAC(…) returns (…) ;let -- specification -- scenarios -- operational profilestel ;
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
40Using Lutess for testing the AC Guiding the generation with operational profiles
Operational profile : Quantitative characterisationOf how the Software will be used
John D. Musa
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
41Using Lutess for testing the AC Guiding the generation with operational profiles
« If the AC is off, the user will (very probably) turn it on »
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
42Using Lutess for testing the AC Guiding the generation with operational profiles
testnode TestAC (…) returns (…) ;var dT : int ;let
environment(…); prob(false -> not(pre(OnOff)), Button, 0.9) ; prob(false -> pre(OnOff), not Button, 0.2) ;tel ;
w.r.t. output
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
43
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
44Using Lutess for testing the AC Guiding the generation with operational profiles
« If the AC is off and the temperature is too high or too low,
then the user will (probably) turn the AC on »
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
45Using Lutess for testing the AC Guiding the generation with operational profiles
prob(false -> not(pre OnOff) and ((Tuser-Tamb > 5) or (Tuser-Tamb < -5)), Button, 0.99) ;
prob(false -> (pre OnOff) or ((Tuser-Tamb < 5) and (Tuser-Tamb > -5)), not Button, 0.01) ; w.r.t. more complex expression
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
46
Lutess
• Random input generation from the environment• Two strategies for guiding• Using CPL
• Similar tool : Lurette– Automata as guides
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
47
Outline
• Lustre / SCADE
• Test generation– Using the description of the environment
– Using the code
• Coverage criteria
• Cost of the test
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
48
Gatel
• Developped at CEA-List in collaboration with IRSN, since 1998, by Bruno Marre et al.
• For automatic generation of test sequencesbased on Lustre/Scade description– Interactive– Based on a CLP interpretation of Lustre/Scade
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
49
Gatel : test generation
• Description in Lustre (A model of) the program under test
• Specification of the environmentto focus on realistic situations
• Test objective to guide the generation
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
50GATeL performs a compilation of Lustre expression into a CLP format
• bool : usual [t,f] domain• int : bounded integers intervals (without modulo)• real: bounded interval arithmetic with double
floating point bounds– real arithmetic– IEEE 754 double floating-point arithmetic
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
51
Test objectives used for test generation
• Reach properties– Properties that must be true at one cycle – To search a test sequence reaching this cycle
• Invariant properties– Properties that is always true– To search for violation in a specified number of steps
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
52
Defining a test objective as a boolean expression
an observation of the past that must be true at the final cycle
(*! reach <bool_expr> !*)
• Reaching an eventno_steam and (pump_level = 50.0)
• Pre-defined scenarioat_cycle(2,u=3) and at_cycle(5,u=6) and at_cycle(7,u=9)
• Declarative scenarioonce_at_least(z) and never_since_last(not(z),x)
• Based on the syntax of an equationCond1 and not Cond2
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
53
Overview of Gatel principles
All possiblebehaviors
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
54
Overview of Gatel principles
Restrictedbehaviors
(environment)
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
55
Overview of Gatel principles
State to be reached
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
56
Overview of Gatel principles
Test sequenceinteractive
construction
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
57
Finding test cases: very simple example
Assert x + y ≥ 10Res = (x ≥ 12) or (y < 13)Reach res=true
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
58
Finding test cases: very simple example
Assert x + y ≥ 10Res = (x ≥ 12) or (y < 13)Reach res=true
y↑
(0,0)
x→10
10Possible valuations
for x and y
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
59
Finding test cases : very simple exampley↑
(0,0)
x→10
10Possible valuations
for x and y
x=14, y=4
Assert x + y ≥ 10Res = (x ≥ 12) or (y < 13)Reach res=true
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
60Finding test cases : reducing solution state spacesdécomposition into 2 subsets w.r.t. “or”
(0,0)
x→10
10
Assert x + y ≥ 10
Res = (x ≥ 12) or (y < 13)
Reach res=true
y↑
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
61Finding test cases : reducing solution state spaces décomposition into 2 subsets w.r.t. “or”
Assert x + y ≥ 10
Res = (x ≥ 12) or (y < 13)
Reach res=true
(0,0)
x→10
10
y↑
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
62Finding test cases : reducing solution state spaces décomposition into 2 subsets w.r.t. “or”
Assert x + y ≥ 10
Res = (x ≥ 12) or (y < 13)
Reach res=true
(0,0)
x→10
10
y↑
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
63
Finding test cases : reducing solution state spaces décomposition into 2 other subsets w.r.t. “≥”
Assert x + y ≥ 10
Res = (x ≥ 12) or (y < 13)
Reach res=true
(0,0)
x→10
10
y↑
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
64
Finding test sequences : simple example
Res = (x ≥ 12) or (y < 13)x + y ≥ 10
x ≥ 12x + y ≥ 10
y < 13x + y ≥ 10
x = 12x + y ≥ 10
x > 12x + y ≥ 10
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
65
Finding test cases : time sequences
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
66
Finding test cases : time sequences
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
67
Finding test cases : time sequences
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
68
Finding test cases : time sequences
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
69
Finding test cases : time sequences
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
70
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
71
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
72
To sum-up with Gatel test generation
• Decomposition of the state space allows to obtain different test cases
• Possibility to define different strategies for splitting solution state space– At the operator level
– At the symbol level
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
73
Splitting the solution setInteractive splitting of the solutions set of a test objective• At the operator level: pre-selected splitting cases on
decision operatorsX = Exp1 and Exp2
– Sequential decomposition1. Exp1 = true ∧ X = Exp22. Exp1 = false ∧ X = false
– Lazy decomposition1. Exp1 = true ∧ Exp2 = true ∧ X = true2. Exp1 = false ∧ X = false3. Exp2 = false ∧ X = false
– Truth table (usual 4 cases)
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
74
Splitting the solution setInteractive splitting of the solutions set of a test objective• At the operator level: pre-selected splitting cases on
decision operators• At the symbol level: user-defined selection strategy attached to
a computed variable(*! split x with [sc1,…,scn] !*)
IntroductionGenerationStoppingEvaluation
LutessGatel
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
75
Outline
• Lustre / SCADE
• Test generation
• Coverage criteria– MTC
– Lustructu
• Evaluation
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
76
SIESTA project: Automation of testing in SCADE based development
Informal requirements
Test objectives
Test specification
Test generation
Test execution
Evaluation
R1R2
C filesE
Scade
Evaluation of research tools for generation
Evaluation of research tools
for coverage / testability
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
77
Using Scade in Avionics : validation problems
R1R2
Covered?
Scade files
Correct?E
S
pre
L1L4
L2
L3
Tests produced • Manually• From requirement • MC/DC coverage
Tracability issuesNo dead code
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
78
Why structural coverage analysis? (DO-248B)
1. Provide evidence that the code structure was verified to the degree required for the applicable software level
2. Provide a means to support demonstration of absence of unintended functions
3. Establish the thoroughness of requirements-based testing”
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
79
Stopping criteria : issues with Lustre
Data-flow : All equations evaluated No graph control
node never (E: bool) returns (S : bool);
let
S = not(E) -> (not(E) and pre(S));
tel;
E
S
pre
L1
L4
L2L3
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
80
• Coverage criteria– adapted to the Lustre paradigm
– measure the test thoroughness and the test effort
• 2 Tools: MTC and LUSTRUCTU– 2 types of criteria
– Based on the network of operators
Structural Coverage Assessment
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
81
MTC : Model Test Coverage
• Esterel Technologies Tool Suite for Scade
• Coverage Criterion based on operator– Each operator has a set of characteristic ‘features’
• Predefined (DC or MCDC)
• Custumized (one feature per qualitatively different behavior)
– Coverage performed for each instance of an operator
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
82
MTC : one criteria for operator
Output = I1 or I2 or I3 or … In
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
83
• defined on the operator network• based on the path activation conditions
ab
lc s
(a = false) or (b = true)(l = true) or (c = false)
⇒ AC = (a = false or b= true) and (l = true or c = false)
Lustructu : an other Coverage Criteria (family)
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
84
• Complexity w.r.t. the path length• Complexity w.r.t. the strength of the criterion
BasicCriterion (BC)
ElementaryConditions
Criterion (ECC)
MultipleConditions
Criterion (MCC)
+
= satisfy the activation conditions
= satisfy all input
variations
= satisfy all edges
variations
a
b
l s
truefalse
c
a
bl1 s
truefalse
c
falsetruel1la
+
Coverage Criteria
IntroductionGenerationStoppingEvaluationv
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
85
LUSTREprogram
Operator networkTest cases
Coverage node 2
...
Coverage node 3
Coverage node n
Execution
Activation condition for P2
...
Activation condition for P3
Coverageratio
Criterionselection
Activation condition for Pn
BC ECC MCC
Lustructu overview
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
86
Example: Alarm Management System (Avionics)
20 nodes 200 operators (boolean, arithmetic, temporal) 300 edges (boolean & integer)
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
87
Activation conditions computation cost
0
5 000
10 000
15 000
20 000
25 000
30 000
35 000
40 000
45 000
50 000
55 000
60 000
65 000
Amount of activation conditions
BC ECC MCC
Criterion strengh
• The stronger the criterion the more the conditions• Affordable cost for small nodes• Increased cost in memory & time for larger nodes
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
88
• Random data: average of 10 test suites• Sequence length: 1 to several thousands
40,00
50,00
60,00
70,00
80,00
90,00
100,00
10 20 30 40 50 60 70 80 90 100 200 300 400 500
Data Sequence Length
Cove
rage
Rat
io (
%)
BC
ECC
MCC
Test effort vs Criteria satisfaction
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
89
Fault detection ability vs mutation
– ~400 mutants for the main node
40,00
50,00
60,00
70,00
80,00
90,00
100,00
10 20 30 40 50 60 70 80 90 100 200 300 400 500
Data Sequence Length
Covera
ge R
ati
o (%
)
BC
ECC
MCC
mutants
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
90
• For nodes with small size - unit test • Methodological suggestions
– First, achieve 100% of BC– Ensure high ratio (70-80%) of ECC and MCC
– 100% of ECC and MCC (if possible)
program analysis + important test effort
Lustructu coverage criteria family: feedback
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
91
• Integration testing
Lustructu current work
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
92
Lustructu perspectives
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
93
• C code coverage– link with the corresponding Lustre code coverage
• Requirements-based coverage [Hemdal Rajan]– criteria defined on formal requirements– generate requirements-based tests
Other approaches for coverage
R1R2
C files
Scade files
E
S
pre
L1L4
L2
L3
IntroductionGenerationStoppingEvaluation
IntroductionMTCLustructu
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
94
Outline
• Lustre / SCADE
• Test generation
• Stopping criteria
• Evaluation
IntroductionGenerationStoppingEvaluation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
95
SIESTA project: Automation of testing in SCADE based development
Informal requirements
Test objectives
Test specification
Test generation
Test execution
Evaluation
R1R2
C filesE
Scade
Evaluation of research tools
for coverage / testability
Evaluation of research tools for generation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
96
Using Scade in Avionics : validation problems
R1R2
Covered?
Scade files
Correct?E
S
pre
L1L4
L2
L3
Tests produced • Manually• From requirement • MC/DC coverage
Tracability issuesNo dead code
IntroductionGenerationStoppingEvaluation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
97
SATAN: a tool for testability analysis
• Tool from LCIS labs, Ch. Robach et al.• For
– Testability analysis (predicting cost of testing)– Location of faults
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
98
Example of Scade program
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
99
Satan : GTI
Lustre to
Information Transfer Graph
IntroductionGenerationStoppingEvaluation
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
100
Information flows
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
101
Information flow to find parts difficult to test
• Information lost in the flow• Compute for each operator
– Controllability– Observability
• Propagated in the flow• Under validation in the project
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
102
Information flow and tracability
• Information flow– One output– Path to input(s)
• Informal requirements – Input and output
• Tracability idea – Linking requirements and information flow through
input and output
• Under work
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
103
Information flow and test objectives for Gatel
• Information flow is a Lustre description• Use it at test objective• Under experimentation
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
104
Tracability
R1R2
Scade files
E
S
pre
L1L4
L2
L3
IntroductionGenerationStoppingEvaluation
IntroductionSatan
Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)
105
Conclusion
Informal requirements
Test objectives
Test specification
Test generation
Test execution
Evaluation
R1R2
C filesE
Scade
Evaluation of Lutess, Gatel
Evaluation of MTC/Lustructu (coverage)
Satan (tracability)
IntroductionGenerationStoppingEvaluation
106
(TAROT 2009)
Testing Lustre/SCADE programs
Lydie du Bousquet, Ioannis Parissis
TAROT Summer School 2009 - July 6-10