Testing TAROT Summer School Lustre/SCADE programs 2009 -...

106
1 (TAROT 2009) Testing Lustre/SCADE programs Lydie du Bousquet, Ioannis Parissis TAROT Summer School 2009 - July 6-10

Transcript of Testing TAROT Summer School Lustre/SCADE programs 2009 -...

Page 1: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

1

(TAROT 2009)

Testing Lustre/SCADE programs

Lydie du Bousquet, Ioannis Parissis

TAROT Summer School 2009 - July 6-10

Page 2: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

2

high level

of criticality

control/command

programs

Synchronous / safety-critical software

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 3: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

3

Reactive systems

Environment system

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 4: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

4

Synchronous program

Synchronous hypothesis : time reaction is null

Environment Program

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 5: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

5

• Esterel, Signal, Lustre

• SCADE (Safety-Critical Applications Development Environment)– Based on Lustre

Synchronous languages

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 6: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

6

SCADE / Lustre overview

• Graphical/textual

• Simple constructions

• Flows and clocks

node never (E: bool) returns (S : bool);

let

S = not(E) -> (not(E) and pre(S));

tel;

E

S

pre

L1

L4

L2L3

E = (E1, E2, E3, … ) S = (S1, S2, S3, … )

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 7: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

7

Example of Lustre program

node never (E: bool) returns (S : bool);

let

S = not(E) -> (not(E) and pre(S));

tel;

clockinput

output

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 8: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

8

Scade V6

• Automaton• Data-flow

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 9: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

9

Using Scade in Avionics : development approach

R1R2

High-level requirements

C-code

C files

Scade files

Low-level requirements

E

S

pre

L1L4

L2

L3

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 10: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

10

Using Scade in Avionics : validation problems

R1R2

Covered?

C files

Scade files

Correct?E

S

pre

L1L4

L2

L3

Correct?

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 11: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

11

Using Scade in Avionics : validation problems

R1R2

Covered?

Scade files

Correct?E

S

pre

L1L4

L2

L3

How to produce tests efficiently ?

Tests produced • Manually• From requirement • MC/DC coverage

Tracability issuesNo dead code

DO – 178 B

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 12: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

12

The SIESTA project (jan 2008 – jan 2011) Automation of testing in SCADE based development

Toulouse Pau

Paris

GrenobleValence

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 13: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

13

SIESTA project: Automation of testing in SCADE based development

Informal requirements

Test objectives

Test specification

Test generation

Test execution

Evaluation

R1R2

C filesE

Scade

Evaluation of research tools for generation

Evaluation of research tools

for coverage / testability

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 14: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

(TAROT 2009)

Objective : automation of the testing process for Scade/Lustre programs

What are the issues ?

Page 15: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

15

Selection of data for Scade/Lustre : issues

Environment Program

(E1, E2, E3, … )

(S1, S2, S3, … )

Test data : sequences of inputs/outputstemporal dependencies (pre, ->, multi-clocks…)

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 16: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

16

Stopping criteria : issues with Lustre

Data-flow and equational: All equations evaluated No graph control

node never (E: bool) returns (S : bool);

let

S = not(E) -> (not(E) and pre(S));

tel;

E

S

pre

L1

L4

L2L3

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 17: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

17

Decision and tracability issues

R1R2

Scade files

E

S

pre

L1L4

L2

L3Informal vs formal

IntroductionGenerationStoppingEvaluation

SynchroneScade/LustreSiestaIssues

Page 18: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

18

Outline

• Lustre / SCADE

• Test generation

• Stopping criteria

• Evaluation

IntroductionGenerationStoppingEvaluation

Page 19: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

19

How to produce test data ?

• Specification / requirement– using the description of its environment– using the specification of the expected behaviours

(test objectives)

• Code

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 20: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

20

Outline

• Lustre / SCADE

• Test generation– Using the description of the environment

– Using the code

• Stopping criteria

• Evaluation

IntroductionGenerationStoppingEvaluation

Page 21: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

21

Environment

Testing using the environment description

Simulation ofthe environment

System

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 22: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

22

Lutess : a tool for testing Lustre programs

input input generatorgenerator

properties

Oracle

Environmentdescription(Lustre)

Programunder test

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 23: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

23

Lutess : a tool for testing Lustre programs (2)

input input generatorgenerator

properties

Programunder test

Oracle

eeee

ssss

e’e’e’e’

ssee

vvLutess V2 : based

on constrain programming)

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 24: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

24

A (very) simple air conditionner controller

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 25: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

25

Lustre implementation of a AC controller

node Program_AC(Button : bool ; Tamb, Tuser : int)returns (OnOff : bool ; Tout : int) ;let OnOff = Button -> ((pre OnOff) and not Button) or (not(pre OnOff) and Button) ; Tout = Tuser + (Tuser - Tamb) / 3 ;tel

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 26: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

26

Behavior of the AC controller

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 27: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

27

Using Lutess for testing the AC

input generatorinput generator

Program_ac.exe

Program under test

testnode TestAC(…) returns (…) ;let -- env. specificationtel ;

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 28: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

28Using Lutess for testing the AC Physical invariants : domain definition

« There are constraints on the temperature range »

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 29: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

29Using Lutess for testing the AC Physical invariants : domain definition

testnode TestAC(OnOff : bool ; Tout : int) returns (Button : bool ; Tamb, Tuser : int) ;let environment(

(Tamb >= -20) and (Tamb <= 60) and (Tuser >= 10) and (Tuser <= 40)) ;tel ;

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 30: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

30

IntroductionGenerationStoppingEvaluation

Page 31: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

31

Page 32: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

32

Using Lutess for testing the AC Physical invariants : system dynamics

« No big variations of the ambient temperature »

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 33: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

33

Using Lutess for testing the AC Physical invariants : system dynamics

testnode TestAC(OnOff : bool ; Tout : int) returns (Button : bool ; Tamb, Tuser : int) ;var dT : int ;let environment(

(Tamb >= -20) and (Tamb <= 60) and (Tuser >= 10) and (Tuser <= 40) and

(dT >= -1) and (dT <= 1) ) ; dT = 0 -> Tamb - pre Tamb ;tel ;

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 34: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

34

Page 35: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

35

Helping the random generation

input generatorinput generator

Program_ac.exe

Program under test

testnode TestAC(…) returns (…) ;let -- specification -- scenarios -- operational profilestel ;

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 36: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

36

Using Lutess for testing the AC Guiding the generation with scenarios

« If the AC is off, then the user will turn it on »

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 37: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

37

Using Lutess for testing the AC Guiding the generation with scenarios

testnode TestAC (…) returns (…) ;var dT : int ;let environment( true->(implies(not(pre(OnOff)), Button) and implies(pre(OnOff), not Button))); … dT = 0 -> Tamb - pre Tamb ;tel ;

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 38: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

38

Page 39: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

39

Helping the random generation

input generatorinput generator

Program_ac.exe

Program under test

testnode TestAC(…) returns (…) ;let -- specification -- scenarios -- operational profilestel ;

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 40: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

40Using Lutess for testing the AC Guiding the generation with operational profiles

Operational profile : Quantitative characterisationOf how the Software will be used

John D. Musa

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 41: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

41Using Lutess for testing the AC Guiding the generation with operational profiles

« If the AC is off, the user will (very probably) turn it on »

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 42: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

42Using Lutess for testing the AC Guiding the generation with operational profiles

testnode TestAC (…) returns (…) ;var dT : int ;let

environment(…); prob(false -> not(pre(OnOff)), Button, 0.9) ; prob(false -> pre(OnOff), not Button, 0.2) ;tel ;

w.r.t. output

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 43: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

43

Page 44: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

44Using Lutess for testing the AC Guiding the generation with operational profiles

« If the AC is off and the temperature is too high or too low, 

then the user will (probably) turn the AC on »

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 45: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

45Using Lutess for testing the AC Guiding the generation with operational profiles

prob(false -> not(pre OnOff) and ((Tuser-Tamb > 5) or (Tuser-Tamb < -5)), Button, 0.99) ;

prob(false -> (pre OnOff) or ((Tuser-Tamb < 5) and (Tuser-Tamb > -5)), not Button, 0.01) ; w.r.t. more complex expression

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 46: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

46

Lutess

• Random input generation from the environment• Two strategies for guiding• Using CPL

• Similar tool : Lurette– Automata as guides

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 47: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

47

Outline

• Lustre / SCADE

• Test generation– Using the description of the environment

– Using the code

• Coverage criteria

• Cost of the test

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 48: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

48

Gatel

• Developped at CEA-List in collaboration with IRSN, since 1998, by Bruno Marre et al.

• For automatic generation of test sequencesbased on Lustre/Scade description– Interactive– Based on a CLP interpretation of Lustre/Scade

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 49: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

49

Gatel : test generation

• Description in Lustre (A model of) the program under test

• Specification of the environmentto focus on realistic situations

• Test objective to guide the generation

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 50: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

50GATeL performs a compilation of Lustre expression into a CLP format

• bool : usual [t,f] domain• int : bounded integers intervals (without modulo)• real: bounded interval arithmetic with double

floating point bounds– real arithmetic– IEEE 754 double floating-point arithmetic

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 51: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

51

Test objectives used for test generation

• Reach properties– Properties that must be true at one cycle – To search a test sequence reaching this cycle

• Invariant properties– Properties that is always true– To search for violation in a specified number of steps

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 52: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

52

Defining a test objective as a boolean expression

an observation of the past that must be true at the final cycle

(*! reach <bool_expr> !*)

• Reaching an eventno_steam and (pump_level = 50.0)

• Pre-defined scenarioat_cycle(2,u=3) and at_cycle(5,u=6) and at_cycle(7,u=9)

• Declarative scenarioonce_at_least(z) and never_since_last(not(z),x)

• Based on the syntax of an equationCond1 and not Cond2

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 53: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

53

Overview of Gatel principles

All possiblebehaviors

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 54: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

54

Overview of Gatel principles

Restrictedbehaviors

(environment)

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 55: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

55

Overview of Gatel principles

State to be reached

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 56: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

56

Overview of Gatel principles

Test sequenceinteractive

construction

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 57: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

57

Finding test cases: very simple example

Assert x + y ≥ 10Res = (x ≥ 12) or (y < 13)Reach res=true

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 58: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

58

Finding test cases: very simple example

Assert x + y ≥ 10Res = (x ≥ 12) or (y < 13)Reach res=true

y↑

(0,0)

x→10

10Possible valuations

for x and y

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 59: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

59

Finding test cases : very simple exampley↑

(0,0)

x→10

10Possible valuations

for x and y

x=14, y=4

Assert x + y ≥ 10Res = (x ≥ 12) or (y < 13)Reach res=true

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 60: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

60Finding test cases : reducing solution state spacesdécomposition into 2 subsets w.r.t. “or”

(0,0)

x→10

10

Assert x + y ≥ 10

Res = (x ≥ 12) or (y < 13)

Reach res=true

y↑

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 61: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

61Finding test cases : reducing solution state spaces décomposition into 2 subsets w.r.t. “or”

Assert x + y ≥ 10

Res = (x ≥ 12) or (y < 13)

Reach res=true

(0,0)

x→10

10

y↑

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 62: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

62Finding test cases : reducing solution state spaces décomposition into 2 subsets w.r.t. “or”

Assert x + y ≥ 10

Res = (x ≥ 12) or (y < 13)

Reach res=true

(0,0)

x→10

10

y↑

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 63: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

63

Finding test cases : reducing solution state spaces décomposition into 2 other subsets w.r.t. “≥”

Assert x + y ≥ 10

Res = (x ≥ 12) or (y < 13)

Reach res=true

(0,0)

x→10

10

y↑

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 64: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

64

Finding test sequences : simple example

Res = (x ≥ 12) or (y < 13)x + y ≥ 10

x ≥ 12x + y ≥ 10

y < 13x + y ≥ 10

x = 12x + y ≥ 10

x > 12x + y ≥ 10

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 65: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

65

Finding test cases : time sequences

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 66: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

66

Finding test cases : time sequences

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 67: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

67

Finding test cases : time sequences

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 68: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

68

Finding test cases : time sequences

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 69: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

69

Finding test cases : time sequences

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 70: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

70

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 71: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

71

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 72: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

72

To sum-up with Gatel test generation

• Decomposition of the state space allows to obtain different test cases

• Possibility to define different strategies for splitting solution state space– At the operator level

– At the symbol level

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 73: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

73

Splitting the solution setInteractive splitting of the solutions set of a test objective• At the operator level: pre-selected splitting cases on

decision operatorsX = Exp1 and Exp2

– Sequential decomposition1. Exp1 = true ∧ X = Exp22. Exp1 = false ∧ X = false

– Lazy decomposition1. Exp1 = true ∧ Exp2 = true ∧ X = true2. Exp1 = false ∧ X = false3. Exp2 = false ∧ X = false

– Truth table (usual 4 cases)

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 74: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

74

Splitting the solution setInteractive splitting of the solutions set of a test objective• At the operator level: pre-selected splitting cases on

decision operators• At the symbol level: user-defined selection strategy attached to

a computed variable(*! split x with [sc1,…,scn] !*)

IntroductionGenerationStoppingEvaluation

LutessGatel

Page 75: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

75

Outline

• Lustre / SCADE

• Test generation

• Coverage criteria– MTC

– Lustructu

• Evaluation

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 76: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

76

SIESTA project: Automation of testing in SCADE based development

Informal requirements

Test objectives

Test specification

Test generation

Test execution

Evaluation

R1R2

C filesE

Scade

Evaluation of research tools for generation

Evaluation of research tools

for coverage / testability

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 77: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

77

Using Scade in Avionics : validation problems

R1R2

Covered?

Scade files

Correct?E

S

pre

L1L4

L2

L3

Tests produced • Manually• From requirement • MC/DC coverage

Tracability issuesNo dead code

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 78: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

78

Why structural coverage analysis? (DO-248B)

1. Provide evidence that the code structure was verified to the degree required for the applicable software level

2. Provide a means to support demonstration of absence of unintended functions

3. Establish the thoroughness of requirements-based testing”

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 79: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

79

Stopping criteria : issues with Lustre

Data-flow : All equations evaluated No graph control

node never (E: bool) returns (S : bool);

let

S = not(E) -> (not(E) and pre(S));

tel;

E

S

pre

L1

L4

L2L3

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 80: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

80

• Coverage criteria– adapted to the Lustre paradigm

– measure the test thoroughness and the test effort

• 2 Tools: MTC and LUSTRUCTU– 2 types of criteria

– Based on the network of operators

Structural Coverage Assessment

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 81: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

81

MTC : Model Test Coverage

• Esterel Technologies Tool Suite for Scade

• Coverage Criterion based on operator– Each operator has a set of characteristic ‘features’

• Predefined (DC or MCDC)

• Custumized (one feature per qualitatively different behavior)

– Coverage performed for each instance of an operator

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 82: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

82

MTC : one criteria for operator

Output = I1 or I2 or I3 or … In

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 83: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

83

• defined on the operator network• based on the path activation conditions

ab

lc s

(a = false) or (b = true)(l = true) or (c = false)

⇒ AC = (a = false or b= true) and (l = true or c = false)

Lustructu : an other Coverage Criteria (family)

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 84: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

84

• Complexity w.r.t. the path length• Complexity w.r.t. the strength of the criterion

BasicCriterion (BC)

ElementaryConditions

Criterion (ECC)

MultipleConditions

Criterion (MCC)

+

= satisfy the activation conditions

= satisfy all input

variations

= satisfy all edges

variations

a

b

l s

truefalse

c

a

bl1 s

truefalse

c

falsetruel1la

+

Coverage Criteria

IntroductionGenerationStoppingEvaluationv

IntroductionMTCLustructu

Page 85: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

85

LUSTREprogram

Operator networkTest cases

Coverage node 2

...

Coverage node 3

Coverage node n

Execution

Activation condition for P2

...

Activation condition for P3

Coverageratio

Criterionselection

Activation condition for Pn

BC ECC MCC

Lustructu overview

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 86: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

86

Example: Alarm Management System (Avionics)

20 nodes 200 operators (boolean, arithmetic, temporal) 300 edges (boolean & integer)

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 87: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

87

Activation conditions computation cost

0

5 000

10 000

15 000

20 000

25 000

30 000

35 000

40 000

45 000

50 000

55 000

60 000

65 000

Amount of activation conditions

BC ECC MCC

Criterion strengh

• The stronger the criterion the more the conditions• Affordable cost for small nodes• Increased cost in memory & time for larger nodes

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 88: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

88

• Random data: average of 10 test suites• Sequence length: 1 to several thousands

40,00

50,00

60,00

70,00

80,00

90,00

100,00

10 20 30 40 50 60 70 80 90 100 200 300 400 500

Data Sequence Length

Cove

rage

 Rat

io  (

%)

BC

ECC

MCC

Test effort vs Criteria satisfaction

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 89: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

89

Fault detection ability vs mutation

– ~400 mutants for the main node

40,00

50,00

60,00

70,00

80,00

90,00

100,00

10 20 30 40 50 60 70 80 90 100 200 300 400 500

Data Sequence Length

Covera

ge R

ati

o (%

)

BC

ECC

MCC

mutants

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 90: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

90

• For nodes with small size - unit test • Methodological suggestions

– First, achieve 100% of BC– Ensure high ratio (70-80%) of ECC and MCC

– 100% of ECC and MCC (if possible)

program analysis + important test effort

Lustructu coverage criteria family: feedback

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 91: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

91

• Integration testing

Lustructu current work

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 92: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

92

Lustructu perspectives

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 93: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

93

• C code coverage– link with the corresponding Lustre code coverage

• Requirements-based coverage [Hemdal Rajan]– criteria defined on formal requirements– generate requirements-based tests

Other approaches for coverage

R1R2

C files

Scade files

E

S

pre

L1L4

L2

L3

IntroductionGenerationStoppingEvaluation

IntroductionMTCLustructu

Page 94: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

94

Outline

• Lustre / SCADE

• Test generation

• Stopping criteria

• Evaluation

IntroductionGenerationStoppingEvaluation

Page 95: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

95

SIESTA project: Automation of testing in SCADE based development

Informal requirements

Test objectives

Test specification

Test generation

Test execution

Evaluation

R1R2

C filesE

Scade

Evaluation of research tools

for coverage / testability

Evaluation of research tools for generation

Page 96: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

96

Using Scade in Avionics : validation problems

R1R2

Covered?

Scade files

Correct?E

S

pre

L1L4

L2

L3

Tests produced • Manually• From requirement • MC/DC coverage

Tracability issuesNo dead code

IntroductionGenerationStoppingEvaluation

Page 97: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

97

SATAN: a tool for testability analysis

• Tool from LCIS labs, Ch. Robach et al.• For

– Testability analysis (predicting cost of testing)– Location of faults

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 98: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

98

Example of Scade program

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 99: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

99

Satan : GTI

Lustre to

Information Transfer Graph

IntroductionGenerationStoppingEvaluation

Page 100: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

100

Information flows

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 101: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

101

Information flow to find parts difficult to test

• Information lost in the flow• Compute for each operator

– Controllability– Observability

• Propagated in the flow• Under validation in the project

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 102: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

102

Information flow and tracability

• Information flow– One output– Path to input(s)

• Informal requirements – Input and output

• Tracability idea – Linking requirements and information flow through

input and output

• Under work

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 103: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

103

Information flow and test objectives for Gatel

• Information flow is a Lustre description• Use it at test objective• Under experimentation

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 104: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

104

Tracability

R1R2

Scade files

E

S

pre

L1L4

L2

L3

IntroductionGenerationStoppingEvaluation

IntroductionSatan

Page 105: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

Lydie du Bousquet, LIG Testing Lustre programs (TAROT’09)

105

Conclusion

Informal requirements

Test objectives

Test specification

Test generation

Test execution

Evaluation

R1R2

C filesE

Scade

Evaluation of Lutess, Gatel

Evaluation of MTC/Lustructu (coverage)

Satan (tracability)

IntroductionGenerationStoppingEvaluation

Page 106: Testing TAROT Summer School Lustre/SCADE programs 2009 - …antares.sip.ucm.es/tarot09/index_files/Du-Bousquet-TAROT... · 2009. 7. 14. · Lydie du Bousquet, LIG Testing Lustre programs

106

(TAROT 2009)

Testing Lustre/SCADE programs

Lydie du Bousquet, Ioannis Parissis

TAROT Summer School 2009 - July 6-10