IT is happening now 2015: Robotprogrammatie (Eric Demeester)

Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen

Eric DemeesterFaculteit Industriële Ingenieurswetenschappen

Technologiecampus Diepenbeek

Onderzoeksgroep ACRO

CEVORA IT event, 25 oktober 2015

Robotprogrammatie: enkele lessen uit de praktijk

Outline

2

• ACRO activities: robots with vision and a plan

o Mobile, assistive robots with vision and a plan

o Agricultural robots with vision and a plan

o Industrial robots with vision and a plan

• Programming robots: learnt lessons

o The RADHAR project as an example

o Requirements

o Solutions

• Trends and challenges for the future


Overview ACRO activities

3

• Profibus & profinet, PLC (UCLL)

o Training for operators in industry

o Service and consultancy regarding industrial networks

• Vision and robotics (KU Leuven)

o Vision: choice of cameras, optics, lighting, image processing

o State estimation and machine learning, e.g.:

• Object recognition/classification and 6D pose estimation

• Mobile robot localisation and mapping

• Estimation of human intentions by robots

o Sensor-based decision making and planning, e.g.:

• Collision-free trajectory planning for mobile robots and industrial manipulators

• Shared human-machine control

Remark: ACRO is part of Faculty of Engineering Technology => research closer to industry (higher TRL levels)


Outline

4







o Requirements

o Solutions



Introduction

5

• Semi-autonomous robotic wheelchairs

o Motivation:

• Many elderly and disabled people suffer from a reduced mobility

• Electric wheelchairs enhance this mobility, but manoeuvring an

electric wheelchair is often difficult and time-consuming

• Consequences: accidents, frustration, dependence on others,

reduced social contact, reduced self-esteem, lower quality of life

o Solution? Equip electric wheelchairs with sensors and

computing power

=> Combine the strengths of both human (global planning) and

computer (fine motion control)

Collaborations/cases with: Permobil, Invacare, HMCI, Ottobock, BlueBotics, National MS Centre

Belgium, Windekind + various research institutes


Research questions

6

• Semi-autonomous robots should be able to answer

three questions:

1. Where am I?


Research questions

7

2. Where am I going?

o Key assumption: user should not be required to

communicate explicitly the desired navigation assistance

o Consequence: user (navigation) plans or intentions are

hidden and should be estimated from uncertain user signals

and sensor signals

o This is the problem of plan (intention) recognition:

Plan recognition is the problem of inferring the goal of

an actor and his plan to achieve this goal, based on a

sequence of actions performed by the actor.


Research questions

8

3. How should I get there?

o Even if it is known where the user would like to drive to, it

remains unclear how the manoeuvre should be executed

jointly by human and robot

o This is the problem of shared control:

The situation in which the control of a device is

shared between one or more users and one or more

robotic controllers.


Research questions

9

o Results with different interfaces: standard joystick, switch

interface, brain-computer interface, haptick joystick


Outline

10







o Requirements

o Solutions



Autonomous apple picking

11

• Introduction:

o It is hard to find personnel for fruit picking

o High wages, low prices for fruit on the global market

o Orchard management: selective spraying/fertilization

• Research questions:

o Autonomously determine the fruit’s position

o Autonomously pick the fruit without damaging the fruit nor the

tree, keeping the stem on the apple

o Feasibility study: is it possible to perform this operation with

vision and a robot arm?

o Can it be performed fast enough?

o And many other ... (logistics, wheather conditions, fruit

selection on the spot, etc.)


Autonomous fruit picking

12

• Current state:

This image cannot currently be displayed.


Outline

13







o Requirements

o Solutions



Random bin picking (IWT Tetra RaPiDo)

14

• Introduction:• Nowadays, objects are typically fed to robots in a well-defined,

mechanised manner, which takes time and money to setup

• Alternative: random bin picking:

• Make a 3D scan of a bin with randomly positioned objects

• Recognise and estimate pose of objects

• Compute a collision-free trajectory taking size of robot, gripper, objects, environment into account

• Research goals:

o Build an open demonstrator

o Evaluate existing open source code

o Focus on integration of vision and robotics

o Speed up of object detection and path planning

o Evaluate potential of novel sensors and algorithms


Random bin picking (IWT Tetra RaPiDo)

15

• Current state:

o Bin picking setup with sheet-of-light working

o Analysis of randomised path generation techniques

Collaborations/cases with: ABB, KUKA, Sick, Materialise, Ceratec, Egemin, cards PLM

solutions, Clock-O-Matic, Dewilde Engineering, Vision++, Exmore Benelux, Flanders Food,

Intrion, Optidrive, PEC, Intermodalics, Rabbit, Robberechts, Robosoft, Robomotive, Sedac

Meral, SoftKinetic, Beltech, Borit, Meditech, Sirris, Phaer,Gibas, Octinion, Dana, Van Hool


Flexible robot welding (IWT VIS SmartFactory)

16

• Introduction:

o Current trends:

• globalisation, individualised products, small lot sizes =>

production industry should become highly flexible

• High wages, lack of skilled labour, global competion => production

industry should become highly automated

• Factory of the future: combine flexibility and automation by using

novel technologies: machine vision, path planning software, force

control, intelligent transport

• Research goals: zero ramp-up & auto-programming• Recognise and determine the pose of objects to be welded using

2D or 3D vision,

• perform quality checks before and after welding

• Computation of optimal (time, energy) trajectories for robots

taking into account: robot geometry, kinematics and dynamics,

singularities, components mounted on the robot, environment


Flexible robot welding (IWT VIS SmartFactory)

17

• Current state:


Vision-based wood waste sorting

18

• Introduction:

o SME innovation project for NV Gielen

o Business model: grind wood waste to wood chips and sell

o Problem: percentage MDF: 15% to 50% => price of wood

chips very low

o Solution: separate MDF from wood

• Research questions:

o How to separate MDF from wood?

• Not possible using only infrared reflection, density, colour

• => classify using vision?

o How to classify it accurately enough?

o How to separate it fast enough?


Vision-based wood waste sorting

19

• Solution:

o Determine several features based on colour, texture, shape

o Train multilayer perceptrions to classify wood and MDF

o Onsorted input (45-55% MDF) � to sorted output with 5%

MDF in wood output and 2% wood in MDF output

o Capacity: 7.5 tons/hour


Outline

20







o Requirements

o Solutions



The RADHAR project as an example

21

Vision: Robotic ADaptation to Humans Adapting to Robots

heterogeneous user groups

with (time)varying skills

dynamic, 3D environments

robots adapting to humans’

signals and responses

Requires life-long

adaptation between

two interacting

learning systems

(human & machine)



22

Consortium

2 user

groupsNMSC, Nationaal

Multiple Sclerosis

Centrum V.Z.W.

Windekind,

school for

children with

disability

3 companies

3

universities

1 research

institute



23

General overview of the framework



24

Developed hardware:


Outline

25







o Requirements

o Solutions

o Problems encountered



Requirements/challenges

26

o Hardware level:

• Be able to deal with different sensors, wheelchairs, interfaces,

people

• Real-time (predictable, fast) control even if data-intensive

o Algorithmic level:

• Easily transfer algorithms to other platforms

• Safety

• Calibration

• Debugging

o Robotics-related challenges:

• How to program the robot to perform what it should do?

• Environment: uncertain/unknown, changing, 3D, soft

• Robot: kinematics and dynamics, slippage, castors, size

• Concurrent tasks/processes/threads


Requirements/challenges

27

o System level: large-scale collaborative research and

development

• Several research groups across Europe develop software

• Portability, different platforms (Windows/Linux, Java/C++)

• Little time during integration weeks (tests on the hardware)

• Runtime robustness: stopping/crashing a module of a running

robot application should not result into an overall

stopping/crashing of the robot application

• Runtime flexibility: be able to change algorithms of modules at

runtime, extend the robot application with new modules at run-

time, probe ingoing and outgoing data of modules at runtime.


Outline

28







o Requirements

o Solutions



Solutions (lessons learnt)

29

1. Hardware abstraction and algorithmic level:

• Use of object-oriented programming (C � C++):

• C++ has more features => more complex, but more readable and less maintenance cost

• C++ might be slightly slower

• Timely reaction, real-time control is very important =>

maintainability, reusability is sometimes ignored (nicely

encapsulated software runs slower)

• => lesson learnt: avoid optimisation until it is needed

• use of GPU � results were not that spectacular (< 10x improvement)

• ... sometimes just use a faster PC



30

• Use of design patterns,

• e.g. to abstract implementation details away from the functionality (e.g. Factory Method): => easily adopt “higher-level” code on different wheelchairs, user interfaces, sensors, ...



31

• Component-based software engineering:

• Compose sw from off-the-shelf and custom-built components

• Well-defined external interface that hides its internals, independently developed from where it is going to be used, clear specification of what it requires and provides and depends on => it can be composed

• 5C’s principle of separation of concerns separating the communication, computation, coordination, configuration, and composition aspects in the overall software functionality. Design patterns exist to decouple these aspects …

- Composition: group entities together, model interactions (= an art)

- Computation: algorithmic part (the “useful” part of the sw)

- Configuration: change settings of a system

- Coordination: how do entities work together, life-cycle FSM

- Communication



32

• Evolution towards a more complex but very structured and motivated Composition Pattern as the basic building block

Vanthienen, Klotzbücher, Bruyninckx, "The 5C-based architectural Composition Pattern: lessons

learned from re-developing the iTaSC framework for constraint-based robot programming", Journal

of Software Engineering for Robotics, pp. 17-35, May 2014.



33

2. Robotics related level:

• Probabilistic robotics: explicitly model uncertainty on all

information sources, and use that uncertainty when taking

decisions

• For intention estimation (plan recognition)

• For shared control


Plan recognition

34

• How to represent intentions?

1. We assume: users wish to reach a certain end pose pe (xe,

ye, θe) with an end velocity ve (ve, ωe)”, e.g.:


Plan recognition

35

2. Furthermore, we assume that users have a certain mental

trajectory in mind to arrive at an end state, e.g.:

i1


Plan recognition

36

• Intention estimation scheme:

1. Generate plan hypotheses i based on local paths or paths to learned end poses or end poses indicated on an a priori map.


Plan recognition

37

2. A probability distribution is maintained over the set of

possible user plans i. Initially, a uniform distribution is

adopted;

3. This distribution is updated every time new user signals uk

are obtained according to Bayes’ rule.


Plan recognition

38

• Example of plan recognition performance:


Shared control

39

• The probability distribution over user plans may be

multi-modal

• Nevertheless, decisions should be made at each

time instant regarding navigation assistance

• 3 approaches have been proposed to make these

decisions:

o Maximum Likelihood (ML)

o Maximum A Posteriori (MAP)

o Greedy Partially Observable Markov Decision Process

(greedy POMDP)


Shared control

40

• Maximum likelihood versus Maximum a posteriori

versus POMDP


Shared control

41

• Example of shared control performance:

o Benchmark test: visit the goal locations 4 – 9 – 6 – 11 – 2 –

14 – 5 – 10 and back to position 4

o Execute this in user control mode, in ML shared control

mode, in POMDP shared control mode

o In shared control, the wheelchair can drive farther than in

user control mode, with the danger of making wrong

decisions and driving too far


Shared control

42

• Example of shared control performance:



43

3. System level:

• we adopted ROS (Robot Operating System), an open-source

middleware initiative that allows:

• Easy and +/- efficient data exchange

• Tools (rosbags) for data collection and replay

• Dynamic configuration of components’ (nodes) parameters

• Interactive GUI for data visualization

• ... And other (multi-language support – C++/Python), support for distributed systems



44

• We developed a module base class, built on top of ROS

• Used to share common functionality between modules (reuse)

• This implemented a finite-state-machine: stop/start a component, be robust against errors, ...

• Health monitor: each component sends a heart-beat message to

let a central controller know about the wellbeing of a component

• The central controller sends a heart-beat to a hardware watchdog

• A GUI allows to display the health state of all modules, to

activate/deactivate modules

• This worked for us, but:

• Required lots of integration work and programming

• Will probably be duplicated by others


Outline

45







o Requirements

o Solutions



Trends and challenges for the future

46

• Probabilistic programming

o Trend towards probabilistic robots (due to many successes)

o Can we incorporate these probabilistic and machine learning

concepts into the programming language?

• E.g. prob distributions as new data type, automatically tune code

using built-in learning functions (program fails => indicate the

correct behaviour and learn from this example)

o Initial successes: much fewer lines of code; e.g. Darpa grand

challenge (autonomous car): 100.000 lines of code

o But: how can it be debugged (randomness plays a crucial

role)



47

• Insurance and liability

o Robots should be safe, e.g. implementing obstacle avoidance

� but how reliable are these? Bugs keep popping up, in

unpredictable way (e.g. memory leaks)

o E.g. by feeding random input or pseudo-random input and

see if the system crashes - a sort of automated unit tests

rather than manually crafted unit tests

• Currently, typically only “nominal” behaviour is evaluated; for

safety “worst-case” behaviour is important

• Reliable communication



48

• Plug-and-play

o there are no standard methods for connecting sensors,

motors, actuators, cameras and other components to robots

• E.g. plug a device in, detect the device, install a driver, know what

can be performed with it, ...

o It’s starting (Orocos, ROS, OpenCV), but this requires still

much programming at lower levels

• Software deployment support:

o Robot “apps” should be made to work easily on different

platforms, have example programs available and good

documentation

Robotprogrammatie: enkele lessen uit de praktijk 49

Dank voor uw aandacht!

Eric Demeester

eric.demeester _at_ kuleuven.be

tel: +32 (0)11 27 88 15

IT is happening now 2015: Robotprogrammatie (Eric Demeester)

Education

Transcript of IT is happening now 2015: Robotprogrammatie (Eric Demeester)