Download - Towards a Benchmark for BPMN Engines-Vincenzo Ferme,Cesare …design.inf.usi.ch/.../wjax2015-Towards_a_Benchmark_for_BPMN_Eng… · Towards a Benchmark for BPMN Engines System’s

Vincenzo Ferme, Cesare Pautasso | University of Lugano (USI)

Towards a Benchmark for BPMN Engines


2

UniversityofStuttgart(IAAS)Germany

UniversityofLugano(USI)Switzerland

Prof. Cesare Pautasso

Vincenzo Ferme

Ana Ivanchikj

Marco Argenti

Gabriele Cerfoglio

Simone D’Avico

Abdil Cakal, Andreia Faria Carvalho

Nicolò Linder, Sonny Monti

Prof. Frank Leymann

Dr. Dieter H. Roller

Marigianna Skouradaki

Abdul Wahab

Nagarjuna Siddam

Tayyaba Azad

Balu Venu Thayil

Credits


3

The Impact of TPC Benchmark on DBMS Performance

Source: http://www.tpc.org/tpcc/results/tpcc_results.asp

http://www.tpc.org/tpcc/results/tpcc_results.asp?print=false&orderby=submitted&sortby=desc


4

What is a Workflow?

Register Reserv.

Send Doc.

Confirm Request


4

What is a Workflow?

Register Reserv.

Send Doc.

Confirm Request

Bob

1


4

What is a Workflow?

Register Reserv.

Send Doc.

Confirm Request

Bob

1

Reservation Service

2

Bob2


4

What is a Workflow?

Register Reserv.

Send Doc.

Confirm Request

Bob

1

Reservation Service

2

Bob2

Alice

3


4

What is a Workflow?

Register Reserv.

Send Doc.

Confirm Request

Bob

1

Reservation Service

2

Bob2

Alice

3

4

Confirmation Handler


Workflow Management System

4

What is a Workflow?

Register Reserv.

Send Doc.

Confirm Request

Bob

1

Reservation Service

2

Bob2

Alice

3

4

Confirmation Handler


Applications

What is a Workflow Management System (WfMS)?

5

WfMS Users

InstanceDatabase

Application Server

DBMS

Web Service

D

A

B

C


6Application Server

Workflow Management System’s Main Components

WES

Job Executor

Core Engine

Application Server


6Application Server


WES

Job Executor

Core Engine

Process Navigator

D

A

BC

Application Server


6Application Server


WES

Job Executor

Core Engine

Process Navigator

D

A

BC

Task Dispatcher Users

Application

Application Server


6Application Server


WES

Job Executor

Core Engine

Process Navigator

D

A

BC


Application

Application Server

Event Handler

Service Invoker Web Service

Application


6Application Server


WES

Job Executor

Core Engine

Process Navigator

D

A

BC

InstanceDatabase

DBMS

Persistent Manager

Transaction Manager


Application

Application Server

Event Handler

Service Invoker Web Service

Application


System’s Architecture• Distributed workflow support • Migrating workflow objects support • Transactional workflow support

Deployment Infrastructure• Standalone • Cluster deployment • Cloud deployment • Mobile deployment

Supported Languages• BPMN, BPEL, Petri-Nets, …

7

Functionality• Dynamic handling of Workflows • Integration capabilities

Workflow Management Systems’ Diversity

BPELEPC XPDL BPMN

1992 1998 2002 2004

PNML

2008

BPMN 2.0

2011





8



BPELEPC XPDL BPMN

1992 1998 2002 2004

PNML

2008

BPMN 2.0

2011





9



BPELEPC XPDL BPMN

1992 1998 2002 2004

PNML

2008

BPMN 2.0

2011





10



BPELEPC XPDL BPMN

1992 1998 2002 2004

PNML

2008

BPMN 2.0

2011


11

https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines

Jan 2011

BPMN 2.0

Jan 2014

BPMN 2.0.2ISO/IEC 19510

Aug 2009

BETABPMN 2.0

Year Number Sum2009 1 12010 5 62011 4 102012 1 112013 8 192014 2 212015 2 23

Grand Total 23

Num

ber o

f BPM

N 2

.0 W

fMSs

0

5

10

15

20

25

Year of the First Version Supporting BPMN 2.0 2009 2010 2011 2012 2013 2014 2015

�1

BPMN 2.0: A Widely Adopted Standard (?)


12

BPMN 2.0


12

B

A

Control Flow

BPMN 2.0


12

Data Flow

A BC

B

A

Control Flow

BPMN 2.0


12

Data Flow

A BC

B

A

Control Flow

Events

BPMN 2.0


12

Data Flow

A BC

B

A

Control Flow

Events Activities Task Types Execution Behaviour

A B

C D

E F

G

H

I

BPMN 2.0


13

Design and implement the first benchmark to assess and compare the performance of WfMSs that are compliant with Business Process Model and Notation 2.0 standard.

”

“

BenchFlow Projectthe goal

http://benchflow.inf.usi.ch


14

end-users, vendors, developersWhy do we Need a Benchmark?


14


1. How to choose the best WfMS in accordance with the company’s technical requirements?

2. How to choose the best WfMS in accordance with the company’s business process models (workflows)? B

CD


14


1. How to choose the best WfMS in accordance with the company’s technical requirements?

2. How to choose the best WfMS in accordance with the company’s business process models (workflows)? B

CD

4. How to identify WfMS’s bottlenecks?

3. How to evaluate performance improvements during WfMS’s development?


15

Workload Mix

Load FunctionsTest Data

Workload Model

WfMSConfigurations

Test Types

Workload Metrics KPIs

Derive

Performance Data

Benchmark Elements for BenchFlow


16

Test Types Load Test, Stress Test

Benchmark Element Limitations

State of the Art Limitations (BPEL WfMS)

[SEKE ’14]Röck et al.


16

Test Types Load Test, Stress TestW

orkl

oad

Mod

el

Workload Mix

Load Functions

Test Data

Unclear Definition

Simplistic

Not Realistic





16


orkl

oad

Mod

el

Workload Mix

Load Functions

Test Data

Unclear Definition

Simplistic

Not Realistic

Syst

ems

Number of WfMS

WfMS Configurations

2–5

Not Extensive





16


orkl

oad

Mod

el

Workload Mix

Load Functions

Test Data

Unclear Definition

Simplistic

Not Realistic

Syst

ems

Number of WfMS

WfMS Configurations

2–5

Not Extensive

Ana

lyse

s

Metrics

KPIs

Too General

Too General





• Relevant

• Representative

• Portable

• Scalable

• Simple

Benchmarking Requirements

• Repeatable

• Vendor-neutral

• Accessible

• Efficient

• Affordable

• K. Huppler, The art of building a good benchmark, 2009 • J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993 • S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A

challenge to software engineering, 2003

References:


18

Main Challenges in Benchmarking BPMN 2.0 WfMSs


18


WORKLOAD MODEL


18


Workload Mix

20%

80%C

A

B

D

WORKLOAD MODEL


18


Workload Mix

20%

80%C

A

B

D

WORKLOAD MODEL

Test Data

40%

60%

CA

B

U S


18


Load FunctionsWorkload Mix

20%

80%C

A

B

D

WORKLOAD MODEL

Test Data

40%

60%

CA

B

U S


18



20%

80%C

A

B

D

BENCHMARK EXECUTION

WORKLOAD MODEL

Test Data

40%

60%

CA

B

U S


18



20%

80%C

A

B

D

WfMS-specific APIs and BPMN 2.0 Customisations

xW

fMS ↔

Use

rs

Clie

nt →

WfM

S

WfM

S ↔

Web

Ser

vice

s

App

licat

ion ↔

WfM

S

BENCHMARK EXECUTION

WORKLOAD MODEL

Test Data

40%

60%

CA

B

U S


18



20%

80%C

A

B

D


xW

fMS ↔

Use

rs

Clie

nt →

WfM

S

WfM

S ↔

Web

Ser

vice

s

App

licat

ion ↔

WfM

S

BENCHMARK EXECUTION

WORKLOAD MODEL

Test Data

40%

60%

CA

B

U S

Asynchronous Execution of Workflows

WfMS

?

start


18



20%

80%C

A

B

D


xW

fMS ↔

Use

rs

Clie

nt →

WfM

S

WfM

S ↔

Web

Ser

vice

s

App

licat

ion ↔

WfM

S

BENCHMARK EXECUTION

WORKLOAD MODEL

ANALYSES

Test Data

40%

60%

CA

B

U S


WfMS

?

start


18


Performance Metrics and KPIs


20%

80%C

A

B

D


xW

fMS ↔

Use

rs

Clie

nt →

WfM

S

WfM

S ↔

Web

Ser

vice

s

App

licat

ion ↔

WfM

S

BENCHMARK EXECUTION

WORKLOAD MODEL

ANALYSES

Test Data

40%

60%

CA

B

U S


WfMS

?

start


19

Define the Workload Mixbased on real-world BPMN 2.0 process models

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest


19


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Obtain Real-World BPMN 2.0 Process ModelsMAIN CHALLENGE:


19


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Obtain Real-World BPMN 2.0 Process ModelsMAIN CHALLENGE:

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

B

A

C


> 100 METRICS:Size, Structure, External Interaction, Data Handling, Complexity

20

Define the Workload Mixcharacterise real-world BPMN 2.0 process models collections

http://benchflow.inf.usi.ch/bpmeter

BPMeter:Web Service and application for static analysis of BPMN 2.0 collections

http://benchflow.inf.usi.ch/bpmeter


21

NUMBER OF WfMSs SUPPORTING THE FEATURE

NU

MB

ER O

F R

EAL-

WO

RLD

MO

DEL

S U

SIN

G T

HE

FEA

TU

RE

MOCK DATA

Define the Workload Mixheterogeneous usage and support of BPMN 2.0


21

NUMBER OF WfMSs SUPPORTING THE FEATURE

NU

MB

ER O

F R

EAL-

WO

RLD

MO

DEL

S U

SIN

G T

HE

FEA

TU

RE

0

200

4

F400

5

600

2

700

8

800

12

950

10

MOCK DATA

Define the Workload Mixheterogeneous usage and support of BPMN 2.0


22

Different Types of Benchmarks

BPMN 2.0 SUPPORTED ELEMENTS

BPM

N 2

.0 A

PPLI

CA

TIO

N D

OM

AIN

CUSTOMCORE ADVANCED FULL

core, advanced, full, custom


22



BPM

N 2

.0 A

PPLI

CA

TIO

N D

OM

AIN



YO

UR

C

OM

PAN

Y


22



BPM

N 2

.0 A

PPLI

CA

TIO

N D

OM

AIN



YO

UR

C

OM

PAN

YD

OM

AIN

SP

ECIF

IC


22



BPM

N 2

.0 A

PPLI

CA

TIO

N D

OM

AIN



YO

UR

C

OM

PAN

YD

OM

AIN

SP

ECIF

ICG

ENER

AL


23

Workload Mix Characterisation Overview

Models Collection

Characterisation

Characterisation Criteria

Benchmark Workflows

Characterise

Selection Criteria


23


Reoccurring Structures

A

H

Models Collection

Characterisation


Benchmark Workflows

Characterise

Selection Criteria


23


Clusters of ModelsReoccurring Structures

A

H

Models Collection

Characterisation


Benchmark Workflows

Characterise

Selection Criteria


24

A

B

CD

H

F

EG

REAL-WORLD PROCESSES

Define the Workload Mixdiscover reoccurring structural patterns


24

A

B

CD

H

F

EG


[SOSE ’15]

a3

a2

a1

a6

a5

a4

Graph Matching

REOCCURRING STRUCTURES

Skouradaki et al.



24

User Defined

Selection Criteria

A

B

CD

H

F

EG


[SOSE ’15]

a3

a2

a1

a6

a5

a4

Graph Matching


Skouradaki et al.



24

User Defined

Selection Criteria

A

B

CD

H

F

EG


WORKLOAD MIX

a1

a2

a1

a3 a550%

50%

Composition Criteria

[SOSE ’15]

a3

a2

a1

a6

a5

a4

Graph Matching


Skouradaki et al.



25

Define the Workload Mixdiscover clusters of models

A

B

CD

H

F

EG



25


A

B

CD

H

F

EG


> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity

CLUSTERS OF MODELS

F

EG

B

CD

A

H

PCA +

Clustering


25


A

B

CD

H

F

EG


> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity

CLUSTERS OF MODELS

F

EG

B

CD

A

H

PCA +

Clustering

WORKLOAD MIX

50%

50%

Clustroid Models

B

CD

A


Define the Test Data

26

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

different types of test data



26

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Test Data:




26

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Test Data:Unsaved diagram

40%

60%Condition Data




26

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Test Data:…

Unsaved diagram

Unsaved diagram

Unsaved diagram

40%

60%Condition Data



27

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Define the Load Functionsdifferent types of load functions


27

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Define the Load Functions

P(branch)

different types of load functions


27

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest


P(branch)

Unsaved diagram

different types of load functions


28


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

Process Logs

process logs, experts’ opinion


28


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

Process Logs

40%

60%

P(branch)



28


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

Process Logs

Completion Time

Completion Time40%

60%

P(branch)



28


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

Process Logs

Response Time

Completion Time

Completion Time40%

60%

P(branch)



28


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

Process Logs

Response Time

Requests RateCompletion Time

Completion Time40%

60%

P(branch)



28


Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

Process Logs

Response Time

Requests RateCompletion Time

Completion Time40%

60%

P(branch)

Experts



29

Test Types and the Workload Model

Performance Test Types

• Load testing

• Stress testing

• Soak testing

• Spike testing

• Scalability testing

• Capacity testing

• Configuration testing

• Isolation testing

• …

Workload Model


29



• Load testing

• Stress testing

• Soak testing

• Spike testing





• …

Workload Model

0

40

80

10 20 30 40 50

Load Function

0

200

400

10 20 30 40 50


29



• Load testing

• Stress testing

• Soak testing

• Spike testing





• …

Workload Model

0

40

80

10 20 30 40 50

Load Function

0

200

400

10 20 30 40 50Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Workload M

ix


29



• Load testing

• Stress testing

• Soak testing

• Spike testing





• …

Workload Model

0

40

80

10 20 30 40 50

Load Function

0

200

400

10 20 30 40 50

Test Data

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Unsaved diagram

RegisterReserv.

SendDoc.

ConfirmRequest

Workload M

ix


30




xW

fMS ↔

Use

rs

Clie

nt →

WfM

S

WfM

S ↔

Web

Ser

vice

s

App

licat

ion ↔

WfM

S

BENCHMARK EXECUTION ANALYSES


20%

80%C

A

B

D

WORKLOAD MODEL

Test Data

40%

60%

CA

B

U S


WfMS

?

start


31

InstanceDatabase

DBMSFaban Drivers

ContainersServers

DATA TRANSFORMERS

ANALYSERS

Performance Metrics

Performance KPIs

harnessWfMS

Test

Exe

cuti

onA

naly

ses

Faban+

Web Service

Enabling the Benchmark Execution and AnalysesBenchFlow framework

Minio


Performance Benchmark

BenchFlow Framework

32

System Under Test (SUT)

core requirements & functionalities



BenchFlow Framework

32

• Automate the SUT deployment

• Simplify the SUT’s deployment configuration




• Manage the benchmark lifecycle

• Simulate the entities interacting with the WfMS

• Accommodate and automate different kinds of performance tests

• Ensure reliable execution

• Ensure repeatability


BenchFlow Framework

32

• Automate the SUT deployment

• Simplify the SUT’s deployment configuration




33

BenchFlow Frameworksystem under test

Docker Engine


33


Docker EngineContainers


33


Docker Engine

Docker Machine

prov

ides

Containers


33


Docker Swarm

Docker Engine

Docker Machine

prov

ides

Containers


33


Docker Swarm

Docker Engine

Docker Machine

prov

ides

manages

Containers

Servers


33


Docker Swarm

Docker Engine

Docker Compose

Docker Machine

prov

ides

manages

Containers

Servers


33


Docker Swarm

Docker Engine

Docker Compose

Docker Machine

prov

ides

manages

Containers

Servers

SUT’s Deployment Conf.

deploys


Manages the Benchmark Lifecycle

BenchFlow Frameworkperformance benchmark

34

Faban




34

Faban

Unsaved diagram

Simulates interacting entities




34

Faban

Unsaved diagram


Ensures repeatability• the behaviour of a load driver

is described using code• load drivers can be released




34

Faban

Unsaved diagram




paired with Docker:SUT’s initial state is frozen in the container




34

Faban

Unsaved diagram


Ensures reliable execution• checks the environment condition• validates the benchmark execution• ensures as few interferences as

possible on the SUT during measurements







34

Faban

Unsaved diagram


Ensures reliable execution• checks the environment condition• validates the benchmark execution• ensures as few interferences as

possible on the SUT during measurements




Automates performance tests


35

software adaptersWfMS-specific APIs

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

Adapters


WfMS-specific APIs

36

WfMS Users

D

A

BC

InstanceDatabase

Application Server

DBMS

Web Service

Load Driver

custom APIs

• Load Driver APIs

• Web Service APIs

• User APIs

• Event APIs

APIs:


37

WfMS-specific APIs requirements from the WfMS

ADVANCED FULLCORE


37


ADVANCED FULLCORE

WfMS

Load Driver APIs

Deploy Workflow

Start Workflow


37


ADVANCED FULLCORE

WfMS

Load Driver APIs

Deploy Workflow

Start Workflow

WfMS

Claim Task Complete Task

User and Web Service APIs

+

Create User Create Group

Pending User Tasks

Invoke WS


37


ADVANCED FULLCORE

WfMS

Load Driver APIs

Deploy Workflow

Start Workflow

+

Event APIs

Issue Event

WfMS

Pending Event Tasks

WfMS

Claim Task Complete Task

User and Web Service APIs

+

Create User Create Group

Pending User Tasks

Invoke WS


38

BPMN 2.0 Customisationsmodels deployment version

WfMS

Deploy Workflow


38


WfMS

Deploy Workflow

BPMN 2.0 Standard XML Serialisation

WfMS A BPMN 2.0

Serialisation

WfMS B BPMN 2.0

Serialisation


38


WfMS

Deploy Workflow


WfMS A BPMN 2.0

Serialisation

WfMS B BPMN 2.0

Serialisation

XML Processor


38


WfMS

Deploy Workflow


WfMS A BPMN 2.0

Serialisation

WfMS B BPMN 2.0

Serialisation

XML Processor

WfMS A WfMS B

|| ||


38


WfMS

Deploy Workflow


WfMS A BPMN 2.0

Serialisation

WfMS B BPMN 2.0

Serialisation

XML Processor

Customised Test Data

Formatting

+WfMS A WfMS B

|| ||


39


WfMS

Start Workflow

overview


Users

D

A

BC

Start

Application Server

Web Service

Load Driver

DBMS

WfMS

InstanceDatabase

39


WfMS

Start Workflow

overview


Users

D

A

BC

Start

Application Server

Web Service

Load Driver

DBMS

WfMS

InstanceDatabase

39


WfMS

Start Workflow

End

overview


40

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

Asynchronous Execution of Workflowsmonitors


40

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

MONITOR

Asynchronous Execution of Workflowsmonitors


41


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

• CPU usage• Database state

Examples of Monitors:Monitors’ Characteristics:• Microservices• Lightweight (written in Go)• As less invasive on the SUT as possible• Expose REST APIs towards the Drivers

monitors


41


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

CPU Monitor

DB Monitor

• CPU usage• Database state

Examples of Monitors:Monitors’ Characteristics:• Microservices• Lightweight (written in Go)• As less invasive on the SUT as possible• Expose REST APIs towards the Drivers

monitors


42




20%

80%C

A

B

D


xW

fMS ↔

Use

rs

Clie

nt →

WfM

S

WfM

S ↔

Web

Ser

vice

s

App

licat

ion ↔

WfM

S

BENCHMARK EXECUTION

WORKLOAD MODEL

ANALYSES

Test Data

40%

60%

CA

B

U S


WfMS

?

start



43

WfMS UsersLoad Driver

InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

levels



43


InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

Metrics and KPIs• Engine Level

levels



43


InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

Metrics and KPIs• Engine Level• Process Level

levels



43


InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

Metrics and KPIs• Engine Level• Process Level• Feature Level

levels



43


InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

Metrics and KPIs• Engine Level• Process Level• Feature Level

• Interactions

levels


44

Performance Metrics and KPIsexamples and relevant users


44

ENGINE LEVEL

PROCESS LEVEL

FEATURE LEVEL



44

Throughput

Response Time

Capacity

Latency

Flexibility to Spike

ENGINE LEVEL

PROCESS LEVEL

FEATURE LEVEL



44

Throughput

Response Time

Capacity

Latency


CPU Usage RAM Usage

Duration of Process

ENGINE LEVEL

PROCESS LEVEL

FEATURE LEVEL



44

Delay of Timer Event

Space Used by a Data Input

Duration of Message Event

Throughput

Response Time

Capacity

Latency


CPU Usage RAM Usage

Duration of Process

ENGINE LEVEL

PROCESS LEVEL

FEATURE LEVEL



44

Delay of Timer Event

Space Used by a Data Input

Duration of Message Event

Throughput

Response Time

Capacity

Latency


CPU Usage RAM Usage

Duration of Process

ENGINE LEVEL

PROCESS LEVEL

FEATURE LEVEL

END-USERS

VENDORS

DEVELOPERS



45

Performance Metrics and KPIsminimal data requirements

Accessibility of the Data

DBMS

WfMS


45


• Workflow & Construct:

• Start Time

• End Time

• [Duration]

Availability of Timing DataAccessibility of the Data

DBMS

WfMS


45


• Workflow & Construct:

• Start Time

• End Time

• [Duration]

Availability of Timing DataAccessibility of the Data

DBMS

WfMS

Availability of Execution StateState of the workflow execution. E.g., running, completed, error



46

collect data

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service



46

collect data

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

MinioInstanceDatabase

Ana

lyse

s

CO

LLECT

OR

S


47

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

• Container’s Stats (e.g., CPU usage)• Database dump• Applications Logs

Examples of Collectors:Collectors’ Characteristics:• Microservices• Lightweight (written in Go)• Two types: online and offline• Buffer data locally• Expose REST APIs towards Drivers

Performance Metrics and KPIscollect data


47

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

Stats Collector

DB Collector

• Container’s Stats (e.g., CPU usage)• Database dump• Applications Logs

Examples of Collectors:Collectors’ Characteristics:• Microservices• Lightweight (written in Go)• Two types: online and offline• Buffer data locally• Expose REST APIs towards Drivers

Performance Metrics and KPIscollect data


48

Performance Metrics and KPIsamount of data

REALISTIC

DATA

Number of

Repetitions

1

1053


48

Performance Metrics and KPIsamount of data

REALISTIC

DATA

Number of

Repetitions

1

1053

Number of WfMSs

1


49

Performance Metrics and KPIstransform data

DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Test

Exe

cuti

on

InstanceDatabase

DATA TRANSFORMERS

Ana

lyse

s

Data Mappers


49


DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Test

Exe

cuti

on

InstanceDatabase

DATA TRANSFORMERS

Ana

lyse

s • custom DB format• unstructured logs• …

Data Mappers


49


DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Test

Exe

cuti

on

InstanceDatabase

DATA TRANSFORMERS

Ana

lyse

s • custom DB format• unstructured logs• …

ETL Process

Data Mappers


50


DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Test

Exe

cuti

on

DATA TRANSFORMERS

Ana

lyse

s

fast and general engine for large-scale data processing

Data Mappers

InstanceDatabase


DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Test

Exe

cuti

on

51

Ana

lyse

s

DATA TRANSFORMERS

ANALYSERS

Performance Metrics

Performance KPIsA

naly

ses

Performance Metrics and KPIsanalyse data

Data Mappers

InstanceDatabase


52

Highlights


52

Highlights

Why a Benchmark for WfMSs


52

Highlights

Why a Benchmark for WfMSs Benchmark’s Main Challenges


52

Highlights

Why a Benchmark for WfMSs Benchmark’s Main Challenges

BenchFlow Framework


53

Future WorkBenchFlow framework

1. Finalise the release of the BenchFlow framework as an open-source project on GitHub and DockerHub;

benchflowhttps://github.com/benchflow


53

Future WorkBenchFlow framework

1. Finalise the release of the BenchFlow framework as an open-source project on GitHub and DockerHub;

benchflowhttps://github.com/benchflow

2. Simplify and automate the execution of common performance tests: load test, spike test, scalability test, …


54

Future Workexperiments

1. Perform the first real-world experiments


54



Execute Different Test Types

• Stress testing

• Spike testing




54




• Stress testing

• Spike testing



Use Real-World Workflows


54




• Stress testing

• Spike testing



Production-like Configurations



54




• Stress testing

• Spike testing



Realistic: • Load Functions • Test Data

Production-like Configurations



55


2. Increase the number of supported WfMSs

APIs: CORE: Load Driver APIs

+ ADVANCED: User and Web Service APIs

+ FULL: Event APIs


55





+ FULL: Event APIs

WfMS N BPMN 2.0

Serialisation

customised test data formatting+

WfMS N

||

Deployable BPMN 2.0 Models:


55





+ FULL: Event APIs

WfMS N BPMN 2.0

Serialisation

customised test data formatting+

WfMS N

||

Deployable BPMN 2.0 Models:

Data: • Accessibility of the data • Availability of timing data • Availability of execution

state


56

•We want to characterise the Workload Mix using Real-World process models

• Share your executable BPMN 2.0 process models, even anonymised

Process Models

• We want to characterise the Load Functions using Real-World behaviours

• Share your execution logs, even anonymised

Execution Logs

Call for CollaborationWfMSs, process models, process logs

•We want to add more and more WfMSs to the benchmark

• Contact us for collaboration, and BenchFlow framework support

WfMSs


Join Us @ ICWE 2016 in Lugano!

http://icwe2016.inf.usi.ch

http://icwe2016.inf.usi.ch

Towards a Benchmark for BPMN EnginesVincenzo Ferme, Cesare Pautasso | University of Lugano (USI)

[email protected]@VincenzoFerme


[email protected]@pautasso

benchflowbenchflow
