Post on 26-Aug-2020
Vincenzo Ferme, Cesare Pautasso | University of Lugano (USI)
Towards a Benchmark for BPMN Engines
Towards a Benchmark for BPMN Engines
2
UniversityofStuttgart(IAAS)Germany
UniversityofLugano(USI)Switzerland
Prof. Cesare Pautasso
Vincenzo Ferme
Ana Ivanchikj
Marco Argenti
Gabriele Cerfoglio
Simone D’Avico
Abdil Cakal, Andreia Faria Carvalho
Nicolò Linder, Sonny Monti
Prof. Frank Leymann
Dr. Dieter H. Roller
Marigianna Skouradaki
Abdul Wahab
Nagarjuna Siddam
Tayyaba Azad
Balu Venu Thayil
Credits
Towards a Benchmark for BPMN Engines
3
The Impact of TPC Benchmark on DBMS Performance
Source: http://www.tpc.org/tpcc/results/tpcc_results.asp
Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register Reserv.
Send Doc.
Confirm Request
Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register Reserv.
Send Doc.
Confirm Request
Bob
1
Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register Reserv.
Send Doc.
Confirm Request
Bob
1
Reservation Service
2
Bob2
Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register Reserv.
Send Doc.
Confirm Request
Bob
1
Reservation Service
2
Bob2
Alice
3
Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register Reserv.
Send Doc.
Confirm Request
Bob
1
Reservation Service
2
Bob2
Alice
3
4
Confirmation Handler
Towards a Benchmark for BPMN Engines
Workflow Management System
4
What is a Workflow?
Register Reserv.
Send Doc.
Confirm Request
Bob
1
Reservation Service
2
Bob2
Alice
3
4
Confirmation Handler
Towards a Benchmark for BPMN Engines
Applications
What is a Workflow Management System (WfMS)?
5
WfMS Users
InstanceDatabase
Application Server
DBMS
Web Service
D
A
B
C
Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job Executor
Core Engine
Application Server
Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job Executor
Core Engine
Process Navigator
D
A
BC
Application Server
Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job Executor
Core Engine
Process Navigator
D
A
BC
Task Dispatcher Users
Application
Application Server
Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job Executor
Core Engine
Process Navigator
D
A
BC
Task Dispatcher Users
Application
Application Server
Event Handler
Service Invoker Web Service
Application
Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job Executor
Core Engine
Process Navigator
D
A
BC
InstanceDatabase
DBMS
Persistent Manager
Transaction Manager
Task Dispatcher Users
Application
Application Server
Event Handler
Service Invoker Web Service
Application
Towards a Benchmark for BPMN Engines
System’s Architecture• Distributed workflow support • Migrating workflow objects support • Transactional workflow support
Deployment Infrastructure• Standalone • Cluster deployment • Cloud deployment • Mobile deployment
Supported Languages• BPMN, BPEL, Petri-Nets, …
7
Functionality• Dynamic handling of Workflows • Integration capabilities
Workflow Management Systems’ Diversity
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
Towards a Benchmark for BPMN Engines
System’s Architecture• Distributed workflow support • Migrating workflow objects support • Transactional workflow support
Deployment Infrastructure• Standalone • Cluster deployment • Cloud deployment • Mobile deployment
Supported Languages• BPMN, BPEL, Petri-Nets, …
8
Functionality• Dynamic handling of Workflows • Integration capabilities
Workflow Management Systems’ Diversity
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
Towards a Benchmark for BPMN Engines
System’s Architecture• Distributed workflow support • Migrating workflow objects support • Transactional workflow support
Deployment Infrastructure• Standalone • Cluster deployment • Cloud deployment • Mobile deployment
Supported Languages• BPMN, BPEL, Petri-Nets, …
9
Workflow Management Systems’ Diversity
Functionality• Dynamic handling of Workflows • Integration capabilities
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
Towards a Benchmark for BPMN Engines
System’s Architecture• Distributed workflow support • Migrating workflow objects support • Transactional workflow support
Deployment Infrastructure• Standalone • Cluster deployment • Cloud deployment • Mobile deployment
Supported Languages• BPMN, BPEL, Petri-Nets, …
10
Workflow Management Systems’ Diversity
Functionality• Dynamic handling of Workflows • Integration capabilities
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
Towards a Benchmark for BPMN Engines
11
https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines
Jan 2011
BPMN 2.0
Jan 2014
BPMN 2.0.2ISO/IEC 19510
Aug 2009
BETABPMN 2.0
Year Number Sum2009 1 12010 5 62011 4 102012 1 112013 8 192014 2 212015 2 23
Grand Total 23
Num
ber o
f BPM
N 2
.0 W
fMSs
0
5
10
15
20
25
Year of the First Version Supporting BPMN 2.0 2009 2010 2011 2012 2013 2014 2015
�1
BPMN 2.0: A Widely Adopted Standard (?)
Towards a Benchmark for BPMN Engines
12
BPMN 2.0
Towards a Benchmark for BPMN Engines
12
B
A
Control Flow
BPMN 2.0
Towards a Benchmark for BPMN Engines
12
Data Flow
A BC
B
A
Control Flow
BPMN 2.0
Towards a Benchmark for BPMN Engines
12
Data Flow
A BC
B
A
Control Flow
Events
BPMN 2.0
Towards a Benchmark for BPMN Engines
12
Data Flow
A BC
B
A
Control Flow
Events Activities Task Types Execution Behaviour
A B
C D
E F
G
H
I
BPMN 2.0
Towards a Benchmark for BPMN Engines
13
Design and implement the first benchmark to assess and compare the performance of WfMSs that are compliant with Business Process Model and Notation 2.0 standard.
”
“
BenchFlow Projectthe goal
http://benchflow.inf.usi.ch
Towards a Benchmark for BPMN Engines
14
end-users, vendors, developersWhy do we Need a Benchmark?
Towards a Benchmark for BPMN Engines
14
end-users, vendors, developersWhy do we Need a Benchmark?
1. How to choose the best WfMS in accordance with the company’s technical requirements?
2. How to choose the best WfMS in accordance with the company’s business process models (workflows)? B
CD
Towards a Benchmark for BPMN Engines
14
end-users, vendors, developersWhy do we Need a Benchmark?
1. How to choose the best WfMS in accordance with the company’s technical requirements?
2. How to choose the best WfMS in accordance with the company’s business process models (workflows)? B
CD
4. How to identify WfMS’s bottlenecks?
3. How to evaluate performance improvements during WfMS’s development?
Towards a Benchmark for BPMN Engines
15
Workload Mix
Load FunctionsTest Data
Workload Model
WfMSConfigurations
Test Types
Workload Metrics KPIs
Derive
Performance Data
Benchmark Elements for BenchFlow
Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress Test
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]Röck et al.
Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress TestW
orkl
oad
Mod
el
Workload Mix
Load Functions
Test Data
Unclear Definition
Simplistic
Not Realistic
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]Röck et al.
Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress TestW
orkl
oad
Mod
el
Workload Mix
Load Functions
Test Data
Unclear Definition
Simplistic
Not Realistic
Syst
ems
Number of WfMS
WfMS Configurations
2–5
Not Extensive
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]Röck et al.
Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress TestW
orkl
oad
Mod
el
Workload Mix
Load Functions
Test Data
Unclear Definition
Simplistic
Not Realistic
Syst
ems
Number of WfMS
WfMS Configurations
2–5
Not Extensive
Ana
lyse
s
Metrics
KPIs
Too General
Too General
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]Röck et al.
Towards a Benchmark for BPMN Engines
• Relevant
• Representative
• Portable
• Scalable
• Simple
Benchmarking Requirements
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009 • J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993 • S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
References:
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
WORKLOAD MODEL
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Workload Mix
20%
80%C
A
B
D
WORKLOAD MODEL
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Workload Mix
20%
80%C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance Metrics and KPIs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
19
Define the Workload Mixbased on real-world BPMN 2.0 process models
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Towards a Benchmark for BPMN Engines
19
Define the Workload Mixbased on real-world BPMN 2.0 process models
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Obtain Real-World BPMN 2.0 Process ModelsMAIN CHALLENGE:
Towards a Benchmark for BPMN Engines
19
Define the Workload Mixbased on real-world BPMN 2.0 process models
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Obtain Real-World BPMN 2.0 Process ModelsMAIN CHALLENGE:
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
B
A
C
Towards a Benchmark for BPMN Engines
> 100 METRICS:Size, Structure, External Interaction, Data Handling, Complexity
20
Define the Workload Mixcharacterise real-world BPMN 2.0 process models collections
http://benchflow.inf.usi.ch/bpmeter
BPMeter:Web Service and application for static analysis of BPMN 2.0 collections
Towards a Benchmark for BPMN Engines
21
NUMBER OF WfMSs SUPPORTING THE FEATURE
NU
MB
ER O
F R
EAL-
WO
RLD
MO
DEL
S U
SIN
G T
HE
FEA
TU
RE
MOCK DATA
Define the Workload Mixheterogeneous usage and support of BPMN 2.0
Towards a Benchmark for BPMN Engines
21
NUMBER OF WfMSs SUPPORTING THE FEATURE
NU
MB
ER O
F R
EAL-
WO
RLD
MO
DEL
S U
SIN
G T
HE
FEA
TU
RE
0
200
4
F400
5
600
2
700
8
800
12
950
10
MOCK DATA
Define the Workload Mixheterogeneous usage and support of BPMN 2.0
Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPM
N 2
.0 A
PPLI
CA
TIO
N D
OM
AIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPM
N 2
.0 A
PPLI
CA
TIO
N D
OM
AIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
YO
UR
C
OM
PAN
Y
Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPM
N 2
.0 A
PPLI
CA
TIO
N D
OM
AIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
YO
UR
C
OM
PAN
YD
OM
AIN
SP
ECIF
IC
Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPM
N 2
.0 A
PPLI
CA
TIO
N D
OM
AIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
YO
UR
C
OM
PAN
YD
OM
AIN
SP
ECIF
ICG
ENER
AL
Towards a Benchmark for BPMN Engines
23
Workload Mix Characterisation Overview
Models Collection
Characterisation
Characterisation Criteria
Benchmark Workflows
Characterise
Selection Criteria
Towards a Benchmark for BPMN Engines
23
Workload Mix Characterisation Overview
Reoccurring Structures
A
H
Models Collection
Characterisation
Characterisation Criteria
Benchmark Workflows
Characterise
Selection Criteria
Towards a Benchmark for BPMN Engines
23
Workload Mix Characterisation Overview
Clusters of ModelsReoccurring Structures
A
H
Models Collection
Characterisation
Characterisation Criteria
Benchmark Workflows
Characterise
Selection Criteria
Towards a Benchmark for BPMN Engines
24
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
Define the Workload Mixdiscover reoccurring structural patterns
Towards a Benchmark for BPMN Engines
24
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph Matching
REOCCURRING STRUCTURES
Skouradaki et al.
Define the Workload Mixdiscover reoccurring structural patterns
Towards a Benchmark for BPMN Engines
24
User Defined
Selection Criteria
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph Matching
REOCCURRING STRUCTURES
Skouradaki et al.
Define the Workload Mixdiscover reoccurring structural patterns
Towards a Benchmark for BPMN Engines
24
User Defined
Selection Criteria
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
WORKLOAD MIX
a1
a2
a1
a3 a550%
50%
Composition Criteria
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph Matching
REOCCURRING STRUCTURES
Skouradaki et al.
Define the Workload Mixdiscover reoccurring structural patterns
Towards a Benchmark for BPMN Engines
25
Define the Workload Mixdiscover clusters of models
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
Towards a Benchmark for BPMN Engines
25
Define the Workload Mixdiscover clusters of models
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity
CLUSTERS OF MODELS
F
EG
B
CD
A
H
PCA +
Clustering
Towards a Benchmark for BPMN Engines
25
Define the Workload Mixdiscover clusters of models
A
B
CD
H
F
EG
REAL-WORLD PROCESSES
> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity
CLUSTERS OF MODELS
F
EG
B
CD
A
H
PCA +
Clustering
WORKLOAD MIX
50%
50%
Clustroid Models
B
CD
A
Towards a Benchmark for BPMN Engines
Define the Test Data
26
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
different types of test data
Towards a Benchmark for BPMN Engines
Define the Test Data
26
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Test Data:
different types of test data
Towards a Benchmark for BPMN Engines
Define the Test Data
26
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Test Data:Unsaved diagram
40%
60%Condition Data
different types of test data
Towards a Benchmark for BPMN Engines
Define the Test Data
26
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Test Data:Unsaved diagram
40%
60%Condition Data
different types of test data
Towards a Benchmark for BPMN Engines
Define the Test Data
26
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Test Data:Unsaved diagram
40%
60%Condition Data
different types of test data
Towards a Benchmark for BPMN Engines
Define the Test Data
26
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Test Data:…
Unsaved diagram
Unsaved diagram
Unsaved diagram
40%
60%Condition Data
different types of test data
Towards a Benchmark for BPMN Engines
27
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Define the Load Functionsdifferent types of load functions
Towards a Benchmark for BPMN Engines
27
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Define the Load Functions
P(branch)
different types of load functions
Towards a Benchmark for BPMN Engines
27
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Define the Load Functions
P(branch)
different types of load functions
Towards a Benchmark for BPMN Engines
27
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Define the Load Functions
P(branch)
different types of load functions
Towards a Benchmark for BPMN Engines
27
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Define the Load Functions
P(branch)
Unsaved diagram
different types of load functions
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
40%
60%
P(branch)
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
Completion Time
Completion Time40%
60%
P(branch)
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
Response Time
Completion Time
Completion Time40%
60%
P(branch)
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
Response Time
Requests RateCompletion Time
Completion Time40%
60%
P(branch)
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
28
Define the Load Functions
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
Process Logs
Response Time
Requests RateCompletion Time
Completion Time40%
60%
P(branch)
Experts
process logs, experts’ opinion
Towards a Benchmark for BPMN Engines
29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Configuration testing
• Isolation testing
• …
Workload Model
Towards a Benchmark for BPMN Engines
29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Configuration testing
• Isolation testing
• …
Workload Model
0
40
80
10 20 30 40 50
Load Function
0
200
400
10 20 30 40 50
Towards a Benchmark for BPMN Engines
29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Configuration testing
• Isolation testing
• …
Workload Model
0
40
80
10 20 30 40 50
Load Function
0
200
400
10 20 30 40 50Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Workload M
ix
Towards a Benchmark for BPMN Engines
29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Configuration testing
• Isolation testing
• …
Workload Model
0
40
80
10 20 30 40 50
Load Function
0
200
400
10 20 30 40 50
Test Data
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Unsaved diagram
RegisterReserv.
SendDoc.
ConfirmRequest
Workload M
ix
Towards a Benchmark for BPMN Engines
30
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance Metrics and KPIs
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION ANALYSES
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
30
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance Metrics and KPIs
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION ANALYSES
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
31
InstanceDatabase
DBMSFaban Drivers
ContainersServers
DATA TRANSFORMERS
ANALYSERS
Performance Metrics
Performance KPIs
harnessWfMS
Test
Exe
cuti
onA
naly
ses
Faban+
Web Service
Enabling the Benchmark Execution and AnalysesBenchFlow framework
Minio
Towards a Benchmark for BPMN Engines
Performance Benchmark
BenchFlow Framework
32
System Under Test (SUT)
core requirements & functionalities
Towards a Benchmark for BPMN Engines
Performance Benchmark
BenchFlow Framework
32
• Automate the SUT deployment
• Simplify the SUT’s deployment configuration
System Under Test (SUT)
core requirements & functionalities
Towards a Benchmark for BPMN Engines
• Manage the benchmark lifecycle
• Simulate the entities interacting with the WfMS
• Accommodate and automate different kinds of performance tests
• Ensure reliable execution
• Ensure repeatability
Performance Benchmark
BenchFlow Framework
32
• Automate the SUT deployment
• Simplify the SUT’s deployment configuration
System Under Test (SUT)
core requirements & functionalities
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker Engine
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker EngineContainers
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker Engine
Docker Machine
prov
ides
Containers
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker Swarm
Docker Engine
Docker Machine
prov
ides
Containers
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker Swarm
Docker Engine
Docker Machine
prov
ides
manages
Containers
Servers
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
prov
ides
manages
Containers
Servers
Towards a Benchmark for BPMN Engines
33
BenchFlow Frameworksystem under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
prov
ides
manages
Containers
Servers
SUT’s Deployment Conf.
deploys
Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Frameworkperformance benchmark
34
Faban
Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Frameworkperformance benchmark
34
Faban
Unsaved diagram
Simulates interacting entities
Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Frameworkperformance benchmark
34
Faban
Unsaved diagram
Simulates interacting entities
Ensures repeatability• the behaviour of a load driver
is described using code• load drivers can be released
Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Frameworkperformance benchmark
34
Faban
Unsaved diagram
Simulates interacting entities
Ensures repeatability• the behaviour of a load driver
is described using code• load drivers can be released
paired with Docker:SUT’s initial state is frozen in the container
Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Frameworkperformance benchmark
34
Faban
Unsaved diagram
Simulates interacting entities
Ensures reliable execution• checks the environment condition• validates the benchmark execution• ensures as few interferences as
possible on the SUT during measurements
Ensures repeatability• the behaviour of a load driver
is described using code• load drivers can be released
paired with Docker:SUT’s initial state is frozen in the container
Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Frameworkperformance benchmark
34
Faban
Unsaved diagram
Simulates interacting entities
Ensures reliable execution• checks the environment condition• validates the benchmark execution• ensures as few interferences as
possible on the SUT during measurements
Ensures repeatability• the behaviour of a load driver
is described using code• load drivers can be released
paired with Docker:SUT’s initial state is frozen in the container
Automates performance tests
Towards a Benchmark for BPMN Engines
35
software adaptersWfMS-specific APIs
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
Adapters
Towards a Benchmark for BPMN Engines
WfMS-specific APIs
36
WfMS Users
D
A
BC
InstanceDatabase
Application Server
DBMS
Web Service
Load Driver
custom APIs
• Load Driver APIs
• Web Service APIs
• User APIs
• Event APIs
APIs:
Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs requirements from the WfMS
ADVANCED FULLCORE
Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs requirements from the WfMS
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs requirements from the WfMS
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
WfMS
Claim Task Complete Task
User and Web Service APIs
+
Create User Create Group
Pending User Tasks
Invoke WS
Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs requirements from the WfMS
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
+
Event APIs
Issue Event
WfMS
Pending Event Tasks
WfMS
Claim Task Complete Task
User and Web Service APIs
+
Create User Create Group
Pending User Tasks
Invoke WS
Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisationsmodels deployment version
WfMS
Deploy Workflow
Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisationsmodels deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard XML Serialisation
WfMS A BPMN 2.0
Serialisation
WfMS B BPMN 2.0
Serialisation
Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisationsmodels deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard XML Serialisation
WfMS A BPMN 2.0
Serialisation
WfMS B BPMN 2.0
Serialisation
XML Processor
Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisationsmodels deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard XML Serialisation
WfMS A BPMN 2.0
Serialisation
WfMS B BPMN 2.0
Serialisation
XML Processor
WfMS A WfMS B
|| ||
Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisationsmodels deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard XML Serialisation
WfMS A BPMN 2.0
Serialisation
WfMS B BPMN 2.0
Serialisation
XML Processor
WfMS A WfMS B
|| ||
Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisationsmodels deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard XML Serialisation
WfMS A BPMN 2.0
Serialisation
WfMS B BPMN 2.0
Serialisation
XML Processor
Customised Test Data
Formatting
+WfMS A WfMS B
|| ||
Towards a Benchmark for BPMN Engines
39
Asynchronous Execution of Workflows
WfMS
Start Workflow
overview
Towards a Benchmark for BPMN Engines
Users
D
A
BC
Start
Application Server
Web Service
Load Driver
DBMS
WfMS
InstanceDatabase
39
Asynchronous Execution of Workflows
WfMS
Start Workflow
overview
Towards a Benchmark for BPMN Engines
Users
D
A
BC
Start
Application Server
Web Service
Load Driver
DBMS
WfMS
InstanceDatabase
39
Asynchronous Execution of Workflows
WfMS
Start Workflow
End
overview
Towards a Benchmark for BPMN Engines
40
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
Asynchronous Execution of Workflowsmonitors
Towards a Benchmark for BPMN Engines
40
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
MONITOR
Asynchronous Execution of Workflowsmonitors
Towards a Benchmark for BPMN Engines
41
Asynchronous Execution of Workflows
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
• CPU usage• Database state
Examples of Monitors:Monitors’ Characteristics:• Microservices• Lightweight (written in Go)• As less invasive on the SUT as possible• Expose REST APIs towards the Drivers
monitors
Towards a Benchmark for BPMN Engines
41
Asynchronous Execution of Workflows
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
CPU Monitor
DB Monitor
• CPU usage• Database state
Examples of Monitors:Monitors’ Characteristics:• Microservices• Lightweight (written in Go)• As less invasive on the SUT as possible• Expose REST APIs towards the Drivers
monitors
Towards a Benchmark for BPMN Engines
42
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance Metrics and KPIs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
42
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance Metrics and KPIs
Load FunctionsWorkload Mix
20%
80%C
A
B
D
WfMS-specific APIs and BPMN 2.0 Customisations
xW
fMS ↔
Use
rs
Clie
nt →
WfM
S
WfM
S ↔
Web
Ser
vice
s
App
licat
ion ↔
WfM
S
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
CA
B
U S
Asynchronous Execution of Workflows
WfMS
?
start
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS UsersLoad Driver
InstanceDatabase
Application Server
DBMS
Web Service
D
A
BC
levels
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS UsersLoad Driver
InstanceDatabase
Application Server
DBMS
Web Service
D
A
BC
Metrics and KPIs• Engine Level
levels
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS UsersLoad Driver
InstanceDatabase
Application Server
DBMS
Web Service
D
A
BC
Metrics and KPIs• Engine Level• Process Level
levels
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS UsersLoad Driver
InstanceDatabase
Application Server
DBMS
Web Service
D
A
BC
Metrics and KPIs• Engine Level• Process Level• Feature Level
levels
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS UsersLoad Driver
InstanceDatabase
Application Server
DBMS
Web Service
D
A
BC
Metrics and KPIs• Engine Level• Process Level• Feature Level
• Interactions
levels
Towards a Benchmark for BPMN Engines
44
Performance Metrics and KPIsexamples and relevant users
Towards a Benchmark for BPMN Engines
44
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIsexamples and relevant users
Towards a Benchmark for BPMN Engines
44
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIsexamples and relevant users
Towards a Benchmark for BPMN Engines
44
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
CPU Usage RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIsexamples and relevant users
Towards a Benchmark for BPMN Engines
44
Delay of Timer Event
Space Used by a Data Input
Duration of Message Event
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
CPU Usage RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIsexamples and relevant users
Towards a Benchmark for BPMN Engines
44
Delay of Timer Event
Space Used by a Data Input
Duration of Message Event
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
CPU Usage RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
END-USERS
VENDORS
DEVELOPERS
Performance Metrics and KPIsexamples and relevant users
Towards a Benchmark for BPMN Engines
45
Performance Metrics and KPIsminimal data requirements
Accessibility of the Data
DBMS
WfMS
Towards a Benchmark for BPMN Engines
45
Performance Metrics and KPIsminimal data requirements
• Workflow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
Towards a Benchmark for BPMN Engines
45
Performance Metrics and KPIsminimal data requirements
• Workflow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
Availability of Execution StateState of the workflow execution. E.g., running, completed, error
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
46
collect data
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
46
collect data
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
MinioInstanceDatabase
Ana
lyse
s
CO
LLECT
OR
S
Towards a Benchmark for BPMN Engines
47
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
• Container’s Stats (e.g., CPU usage)• Database dump• Applications Logs
Examples of Collectors:Collectors’ Characteristics:• Microservices• Lightweight (written in Go)• Two types: online and offline• Buffer data locally• Expose REST APIs towards Drivers
Performance Metrics and KPIscollect data
Towards a Benchmark for BPMN Engines
47
DBMSFaban Drivers
ContainersServers
harnessWfMS
Test
Exe
cuti
on Web Service
Stats Collector
DB Collector
• Container’s Stats (e.g., CPU usage)• Database dump• Applications Logs
Examples of Collectors:Collectors’ Characteristics:• Microservices• Lightweight (written in Go)• Two types: online and offline• Buffer data locally• Expose REST APIs towards Drivers
Performance Metrics and KPIscollect data
Towards a Benchmark for BPMN Engines
48
Performance Metrics and KPIsamount of data
REALISTIC
DATA
Number of
Repetitions
1
1053
Towards a Benchmark for BPMN Engines
48
Performance Metrics and KPIsamount of data
REALISTIC
DATA
Number of
Repetitions
1
1053
Number of WfMSs
1
Towards a Benchmark for BPMN Engines
49
Performance Metrics and KPIstransform data
DBMSFaban Drivers
ContainersServers
harnessWfMS
Web Service
Test
Exe
cuti
on
InstanceDatabase
DATA TRANSFORMERS
Ana
lyse
s
Data Mappers
Towards a Benchmark for BPMN Engines
49
Performance Metrics and KPIstransform data
DBMSFaban Drivers
ContainersServers
harnessWfMS
Web Service
Test
Exe
cuti
on
InstanceDatabase
DATA TRANSFORMERS
Ana
lyse
s • custom DB format• unstructured logs• …
Data Mappers
Towards a Benchmark for BPMN Engines
49
Performance Metrics and KPIstransform data
DBMSFaban Drivers
ContainersServers
harnessWfMS
Web Service
Test
Exe
cuti
on
InstanceDatabase
DATA TRANSFORMERS
Ana
lyse
s • custom DB format• unstructured logs• …
ETL Process
Data Mappers
Towards a Benchmark for BPMN Engines
50
Performance Metrics and KPIstransform data
DBMSFaban Drivers
ContainersServers
harnessWfMS
Web Service
Test
Exe
cuti
on
DATA TRANSFORMERS
Ana
lyse
s
fast and general engine for large-scale data processing
Data Mappers
InstanceDatabase
Towards a Benchmark for BPMN Engines
DBMSFaban Drivers
ContainersServers
harnessWfMS
Web Service
Test
Exe
cuti
on
51
Ana
lyse
s
DATA TRANSFORMERS
ANALYSERS
Performance Metrics
Performance KPIsA
naly
ses
Performance Metrics and KPIsanalyse data
Data Mappers
InstanceDatabase
Towards a Benchmark for BPMN Engines
52
Highlights
Towards a Benchmark for BPMN Engines
52
Highlights
Why a Benchmark for WfMSs
Towards a Benchmark for BPMN Engines
52
Highlights
Why a Benchmark for WfMSs Benchmark’s Main Challenges
Towards a Benchmark for BPMN Engines
52
Highlights
Why a Benchmark for WfMSs Benchmark’s Main Challenges
BenchFlow Framework
Towards a Benchmark for BPMN Engines
53
Future WorkBenchFlow framework
1. Finalise the release of the BenchFlow framework as an open-source project on GitHub and DockerHub;
benchflowhttps://github.com/benchflow
Towards a Benchmark for BPMN Engines
53
Future WorkBenchFlow framework
1. Finalise the release of the BenchFlow framework as an open-source project on GitHub and DockerHub;
benchflowhttps://github.com/benchflow
2. Simplify and automate the execution of common performance tests: load test, spike test, scalability test, …
Towards a Benchmark for BPMN Engines
54
Future Workexperiments
1. Perform the first real-world experiments
Towards a Benchmark for BPMN Engines
54
Future Workexperiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Towards a Benchmark for BPMN Engines
54
Future Workexperiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Use Real-World Workflows
Towards a Benchmark for BPMN Engines
54
Future Workexperiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Production-like Configurations
Use Real-World Workflows
Towards a Benchmark for BPMN Engines
54
Future Workexperiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Realistic: • Load Functions • Test Data
Production-like Configurations
Use Real-World Workflows
Towards a Benchmark for BPMN Engines
55
Future Workexperiments
2. Increase the number of supported WfMSs
APIs: CORE: Load Driver APIs
+ ADVANCED: User and Web Service APIs
+ FULL: Event APIs
Towards a Benchmark for BPMN Engines
55
Future Workexperiments
2. Increase the number of supported WfMSs
APIs: CORE: Load Driver APIs
+ ADVANCED: User and Web Service APIs
+ FULL: Event APIs
WfMS N BPMN 2.0
Serialisation
customised test data formatting+
WfMS N
||
Deployable BPMN 2.0 Models:
Towards a Benchmark for BPMN Engines
55
Future Workexperiments
2. Increase the number of supported WfMSs
APIs: CORE: Load Driver APIs
+ ADVANCED: User and Web Service APIs
+ FULL: Event APIs
WfMS N BPMN 2.0
Serialisation
customised test data formatting+
WfMS N
||
Deployable BPMN 2.0 Models:
Data: • Accessibility of the data • Availability of timing data • Availability of execution
state
Towards a Benchmark for BPMN Engines
56
•We want to characterise the Workload Mix using Real-World process models
• Share your executable BPMN 2.0 process models, even anonymised
Process Models
• We want to characterise the Load Functions using Real-World behaviours
• Share your execution logs, even anonymised
Execution Logs
Call for CollaborationWfMSs, process models, process logs
•We want to add more and more WfMSs to the benchmark
• Contact us for collaboration, and BenchFlow framework support
WfMSs
Towards a Benchmark for BPMN Engines
Join Us @ ICWE 2016 in Lugano!
http://icwe2016.inf.usi.ch
Towards a Benchmark for BPMN EnginesVincenzo Ferme, Cesare Pautasso | University of Lugano (USI)
vincenzo.ferme@usi.ch@VincenzoFerme
http://benchflow.inf.usi.ch
c.pautasso@ieee.org@pautasso
benchflowbenchflow