Karel de Grote University College

Luc Mertens + Rudi Penne + Wim Abbeloos

Master Electromechanics, Industrial Automation

Studenten: Masters EMAU + Erasmus + Projecten …

Research: Tetra, KHLim, Phaer, DataVision, Egemin, Hero, IFM, …

September 2012 www.innovatienetwerk.be/projects/1583

Industrial

Luc.mertens@kdg.be

wim.abbeloos@kdg.be

Innovatie door

Wetenschap en Technologie

03D2xx-cameras PMD[Vision]CamCube 3.0 Fotonic RGB_Z70

van IFM-Electronics Highest Res = 352 x 288 pix! 160 x 120 pixel

(64x50 pix.)

Swiss Ranger MESA 170 x 145 pix. DepthSense 320

( 320 x 240 pix )

‘Time of Flight’ camera types .

Melexis: EVK75301 80x60 pix.

j

i

I

1 1

J

u.ku

v.kv

d

r D

x

y

z

(0,0,f)

f

R

phi

u = j – J/2 ; uk = ku*u v = i – I/2 ; vk = kv*v tg φ = uk/f r = √(xu²+f²) d = √(uk²+vk²+f²) D/d = x/uk = y/vk = z/f

J/2

I/2

ToF VISION: world to image, image to world - conversion

N

Aan elk wereldpunt en aan zijn locale omgeving kunnen 12 belangrijke coördinaten verbonden worden:

x, y, z, Nx, Ny, Nz, N, kr, kc, R, G, B .

kr ,kc

A

f

a P

vt

yPc

zPc

zw

yw

1. Camera x // World x // Robot x

2. World (yw,zw) = Robot (yr,zr) + ty

3. yPw = -(z0 – zPc).sin(a) + yPc.cos(a) = yPr + ty

4. zPw = (z0 – zPc).cos(a) + yPc.sin(a) = zPr

a

Material handling

Work plane = reference.

yr

zr

ty

Mitsubishi

Pepper handling

Connected peppers can be

distinguised by means of local

gradients. Gradients can

quickly be direved from

d/D-ratios.

'PepperD_short.m'.

Stitching two or more ToF-images.

Double the angle of view / generalized binocular view.

z0

Z

zR

zL DL DR

xR

X

z0 is the intersection distance of both symmetrical optical axes.

XL = xL.cos(a) – (zL+z0).sin(a)

XR = xR.cos(a) + (zR+z0).sin(a)

ZL = xL.sin(a) + (zL+z0).cos(a)

ZR = -xR.sin(a) + (zL+z0).cos(a)

a- a+

Straightforward calibration.

Options: - alternated image triggering (left/right)

- simultaneous triggering but with different modulation frequencies .

Cameleon View

Other experiments with ToF cameras

Real images of

a sewer at KdG.

Stitched volume

(6x7 images).

Dimensions

in meter.

Stitching ToF-images.

Outdoor ToF experiments: road quality analysis. VIM.m

www.TriDiCam.net

Line sensor

The first line sensor with a

resolution of 64 x 2 pixel

to be developed by

Fraunhofer IMS.

15 kHz scan speed.

3 mm x 10000 Hz 30 m/s

108 km/u @ 10000 Hz 3 mm.

Pixel shape

Motion Blur

- deblurring is structural incompatible with noise (see: Wiener Filters) !

- the unstability can be explained from matrix algebra (patent iVL).

- instability reduction is possible to some extend (regularisation).

- the better we can handle ‘blur’ in real time the faster objects may

flow ‘through’ the ToF-images. (VIM-project).

- Motion Blur in ToF-images needs further investigations… PhD ?

Which pixels have ‘seen’ a

specific dice eye and for what

integration time.

Motion deblur for ‘color

cameras ’ and for ‘ToF cameras’

ask for different approaches.

Speed: v = 100 mm/sec ; FoV = 100 pixel/1000 mm.

Pixel speed = v*FoV = 10 pixel/sec .

Sample rate: f = 10 Hz ~ Edge displacement = 1 pixel/sample.

z

y = v.t

Closer to the

ToF camera

Further away

Closer

Further away

from the ToF

camera

Speed evaluation of a ToF camera 1/2.

Blurred v.Ti

Linear displacement during

the integration time Ti :

Δs = v.Ti mm.

e.g. FoV = 100 pixel / 1000 mm. Linear edge displacement:

Δp = Δs . FoV pixel.

‘Moving edge’-

detection

v

Speed: ω = 1 rad/sec (± 60°/sec) ; Radius = 300 mm. Factor = 50 pixel/600 mm.

Tip speed: Vmax = ω.Rmax = 300 mm/sec. Tip pixel speed = V*Factor = 25 pixel/sec .

Sample rate: f = 10Hz ~ Tip displacement = 30 mm/sample ~ 2.5 tip pixel/sample.

z

ω.t

360° 180° 0°

Closer to the

ToF camera

Further away

Closer

Further away

from the ToF

camera

Speed evaluation of a ToF camera 2/2.

Blurred ω.Ti

Angular displacement during

the integration time Ti :

Δα = ω.Ti rad.

Tangential pixel displacement:

Δp = R.Δα.Factor pixel.

Spinning Wheel

Tetra: time to bridge

RGBd, ToF !

European experts

PMD-technologies

SoftKinetic

MESA

ifm electronic

Fotonic

Canesta

Zess Uni-Siegen

DAGM

VISICS

ETH Zurich

MUN

Metrilis

Vision++

TriDiCAM

Omec Beckon

European Projects:

Artts

Lynkeus

3D4YOU

Moses

IURO

… … … …

Who can use it ?

Who is waiting for it ?

Robotics

Navigation (AGV, Wheelchair)

Automotive

Material Handling

Automated Harvesting

Industrial Production

Bin Picking

Gesture Analysis

Human Machine Interfaces

Domotics

Gaming

Augmented reality

Security

Healthcare

Supervision on elderly people

Industry – SME

Teachers – Students

Professionals … … …

2.d = λmod . [φ/2π] ___

Error = k. λmod / (√ n . S/N )

Light takes about 66 picoseconds to travel over 1 cm.

λmod = 15 meter ; fmod = 20 MHz

λmod = 10 meter ; fmod = 30 MHz

Time of Flight

Cameras

c = λmod * fmod

λNIR = +/- µm.

Photonic Mixing Device PMD

A

Lambert : I(θ) = Imax.cos(θ) and Reflection factor BRDF (1965)

BRDF stands for: ‘bidirectional reflectance distribution function’

Each point of a surface has a normal direction n. Incident light from a direction ωin is scattered in all directions ωscatter . BRDF describes the fraction f(ωin , ωscatter) between incident and scattered light energy.

‘Lambertian’ reflection law Random texture

ωscatter

ωin

Time of Flight Cameras: error types

External error factors:

- surfaces with a low reflectivity (e.g. black and/or rough objects,..)

- unexpected NIR light sources (e.g. sun light, heated objects, ..)

- a low angle of incident ( e.g. the walls in a tunnel, .. ).

- saturation effects caused by bright reflections ( e.g. plastic, metals,.. ).

- scattering and absorption by dust, smoke or fog.

- Multiple reflections – Multi path .

- Ambiguity range (‘back-folding’).

Internal error factors:

- non-harmonic and/or inhomogeneous LED light (spatial and over time).

- gamma factor & saturation effects of CMOS (non linear integration).

- shot noise (specle), discretization noise, dark currents.

- distortions and blur caused by lenses or moving objects.

f = 80.0

f = 140.0

A simple calibration step gives the intrinsic ToF-camera parameters: ku, kv, f, ( + distorsion parameters ).

v

u

The calibration of a flat plane: e.g. a robot work plane

Resizing

+ filtering.

Temporal,

spatial

filtering.

Calibration: based on the

perception of a flat plane:

• Select a rough substrate.

• Evaluate the row and column wise

standard deviations as a function of

the integration time.

• Eliminate (if present) the systematic

odd-even-errors.

• Make use of an adequate filter:

Gaussian, Mean, Median, …

• Calculate the best fit reference

plane: z = a + b.x + c.y .

• Find the camera rotation angles:

ROTx and ROTy .

• Find the central distance z0.

Calibration = shoot some images,

calculate the parameters…

At that moment the camera is

ready for use: … ToF !

f , ku , kv

z

nx ~ f.(D4/d4 - D3/d3)/(D4/d4 + D3/d3)

ny ~ f.(D2/d2 - D1/d1)/(D2/d2 + D1/d1)

nz ~ - (u.nx + v.ny + 1 ) ▪

n d

D

d² = u² + v² + f².

O

3

1

4

2

The world normal vector at a

random position: n (nx, ny and nz) .

D/d-related calculations. Fast calculations !!

Nx,Ny ~ d(D/d)x,y / (D/d)

f

u

v

Second differences.

D-related calculations. Rapid calculations !!

Fast flatness analysis.

D1/d1 = X1/j1 ; D2/d2 = X2/j2 ; D3/d3 = X3/j3 . Take e.g.: X1= X2 = X3

- D1 + 2D2 - D3 = ( -d1/j1 + 2d2/j2 - d3/j3 ) X1 . Del2(D) ≠ 0 !

-D1 + 2D2 - D3 = [ (-1/sin1 + 1/sin2) + (1/sin2 - 1/sin3 )] X1 ≠ 0 ! Del2(d/D) =

0 !

≠

≠

f

j1 j2 j3

d1 d2 d3

D1

D2

D3

x

z

f

j1 j2 j3 j4 j5

d1 d2 d3 d4

D1

D2

D3

D4

x

z AC/AD ρ = --------

BC/BD

(j1-j4)/(j1-j5)

ρ = -------------- = 9/8

(j2-j5)/(j2-j5)

x1

Cross Ratios.

Top View Top View

A

B

C

D

Cross Ratios are

invariant under

projections !!

D1² + D3² - 2.D1.D3.cos(D1,D3) D1/D3 + D3/D1 – 2.cos(D1,D3)

D1² + D4² - 2.D1.D4.cos(D1,D4) D1/D4 + D4/D1 – 2.cos(D1,D4)

=

D2² + D3² - 2.D2.D3.cos(D2,D3) D3/D2 + D2/D3 – 2.cos(D2,D3)

D2² + D4² - 2.D2.D4.cos(D2,D4) D2/D4 + D4/D2 – 2.cos(D4,D2)

ρ² =

1 2

3 4

2.cos(D1,D3) = 2.cos(d1,d3) = [ d1² + d3² - (j3-j1)² ] / d1d3 2.cos(D1,D4) = 2.cos(d1,d4) = [ d1² + d4² - (j4-j1)² ] / d1d4

2.cos(D2,D3) = 2.cos(d2,d3) = [ d2² + d3² - (j3-j2)² ] / d2d3

2.cos(D2,D4) = 2.cos(d2,d4) = [ d2² + d4² - (j4-j2)² ] / d2d4

Cross Ratios in practice.

D1

D2

D3

D4

d4

d2 d3

d1

j1 j2

j3 j4

[D1//D3 + D3/D1 – 2.cos(d1,d3) ].[D2//D4 + D4/D2 – 2.cos(d2,d4) ]

[D1//D4 + D4/D1 – 2.cos(d1,d4) ].[D2//D3 + D3/D2 – 2.cos(d2,d3) ]

ρ² =

sensor

Optical centre

ρ² = (j3 – j1)²/(j4 – j1)²

(j3 – j2)²/(j4 – j2)² = (9/16)/(4/9) = 81/64

ρ = 9/8 For equidistant pixels:

Fast Flatness Analysis

The cosine values can be calculated in advance !

d/D-related calculations. Rapid calculations !!

Collinearity and coplanarity: ToF !

d1/D1 - d2/D2 + d3/D3 - d4/D4 = 0 (Coplanarity constraint for regular grids).

(d1/D1).[(u2v3 – u3v2) + (u3v4 – u4v3) + (u4v2 – u2v4)] -

(d2/D2).[(u1v3 – u3v1) + (u3v4 – u4v3) + (u4v1 – u1v4)] + (Coplanarity

(d3/D3).[(u1v2 – u2v1) + (u2v4 – u4v2) + (u4v1 – u1v4)] - Constraint).

(d4/D4).[(u1v2 – u2v1) + (u2v3 – u3v2) + (u3v1 – u1v3)] = 0

d1/D1 – 2.(d2/D2) + d3/D3 = 0 . (Collinearity Constraint for a regular grid:

del2(d/D) = 0 ! u1 = u2-1 ; u3 = u2+1 ).

?

d/D-related calculations. Rapid calculations !!

Coplanarity based calibration = finding f , corr, kv and ku .

d1/D1 + d3/D3 = d2/D2 + d4/D4 (Coplanarity constraint for regular grids

Dimensionless variants are possible ).

1

2

3 4

Coplanarity constraint for regular grids

d1/D1 + d3/D3 = d2/D2 + d4/D4 ( row / column balance ). dD = d./D; filter = [0 0 1 0 0 ; 0 0 0 0 0 ; -1 0 0 0 -1 ; 0 0 0 0 0 ; 0 0 1 0 0]; Flat = conv2(dD,filter,'same'); bwFlat = abs(Flat) < 1 & ~bwFloor;

Note: for a color camera when three points are 3D-located a fourth point is bounded if it is known to be in the same plane.

d/D-related calculations. Rapid calculations !!

MESA SR 4000.

Coplanarity check in a ToF image: fast and easy .

Wall

Floor

Wooden plates

Corridor 20 m.

D/d-related calculations.

For navigation purposes, the free floor area can be found from:

di/Di = e/E = [ f.sin(a0) + vi.cos(a0) ] / E .

The points near the floor fulfil the balance: E = (Di/di)[ f.sin(a0)+ vi.cos(a0)] .

Note: for a color camera ‘floor points’ fulfil: Di = E.di/[ f.sin(a0)+ vi.cos(a0)]

a0

vi

f

E di

e

Di

Camera bounded

parallel to the floor.

Floor.

camera

zi

Camera

inclination = a0 .

MESA SR4000

Case:

Bin picking

H E R O X

HERO X

Time of Flight camera

2011

YouTube: KdGiVL

Bin picking & 3D-OCR

Analyse ‘blobs’ one by one.

Find the centre of gravity XYZ, the normal direction components Nx, Ny, Nz ,

the so called ‘Tool Centre Point’ and the WPS coordinates.

YouTube: KdGiVL

Basic tasks of ToF cameras in order to support Healthcare Applications:

• Guide an autonomous wheelchair along the wall of a corridor.

• Avoid collisions between an AGV and unexpected objects.

• Give warnings about obstacles (‘mind the step’…, ‘kids’, stairs…)

• Take position in front of at a table, a desk or a TV monitor.

• Drive over a ramp at a front door or the door leading to the garden.

• Drive backwards based on a ToF camera (find the free floor).

• Park a wheelchair or an AGV autonomous.

• Command a wheelchair to reach a given position.

• Guide a patient in a semi automatic bad room.

• Support the supervision over elderly people.

• Fall detection of (elderly) people.

Healthcare Center (UA). Bart Geraets & Jan Steckel.

Imagine a wheelchair user likes to do 8 typical activities at home:

1. Sitting on the table. 2. Watching TV. 3. Looking through the window.

4. Working on a PC. 5. Reaching the corridor 6. Command to park the

wheelchair. 7. Finding some books on a shelf. 8. Reaching the kitchen

and the garden.

MESA

SR 4000

Potential driven navigation

- a known map

- ‘square waves’

- distance to end point

conform to map constraints

blue = end point

- a known map

- distance to nearest object

conform to object constraints

blue = nearby

Target point No local minima Repulsive potential

Path finder based on

ToF measurements &

guided potential flow.

Path building is possible

by means of the

‘Dijkstra Algorithm’.

ToF - RGB

correspondency

x

y

IDS uEye

UI-1240SE-C

x y z

O3D2xx MESA SR4000

vc,P/F – kv.vt/f = tx.√(z²P+y²P)

uc/F = ku.ut/F .

Other experiments with ToF cameras (1)

1. The ‘region of interest (ROI)’ is the floor ( … and not the garbage behind it ).

2. Since the camera is inclined (Rotx, Roty) one single ToF image

can fit the floor as:

z = a + b.x + c.y ( one image, CPUtime = 1 second ).

3. The surface of every strange object on the floor is covered by:

z < a + b.x + c.y

4. To become a stable black/white image a threshold is used:

bw = ( z + threshold < a + b.x + c.y ) . MESA SR4000

IDS uEye

UI-1240SE-C

Other experiments with ToF cameras (2)

Palette

Angles can be measured relatively

precise… use this fact if possible.

An equilibration of the angle errors is

also possible.

Automated ‘palette handling’..

using Time of Flight !

Desired accuracy:

1.5 cm absolute error on a

distance of 1.5 meter.

RGBd can be used as a

’ sub pixel GONIOMETER ! ‘

Muliti path !

Other experiments with ToF cameras (3)

ifm O3D2xx ToF 64 x 50 pixel 248 x 192

Inspect a random stack of ‘Palettes’

1 jL jM J2 jR J (Columns from 1:J)

D f

β0

β1 β2

tg(β0) = (jM-J2)/f tg(β0+β1) = (jL-J2)/f and tg(β0-β2) = (jR-J2)/f

β0 = atan2(jM-J2,f) β1 = atan2(jL-J2,f) - β0 and β2 = β0 + atan2(jR-J2,f)

cotg(α) = x/y = sin(β2–β1)/(2.sin(β1).sin(β2)) = tg(π/2-α)

x = D.cos(α) ; y = D.sin(α)

ToF searches at a lower resolution,

RGB is able to correct.

x

y

RGBd cameras

DepthSense 311

1. Find the z –discontinuity

2. Look for vertical and

forward oriented regions

3. Check the collineraity

4. Use geometrical laws

in order to find

x, y, z and b.

RGBd

Mesa SwissRanger SR4000

1. Remove bright pixels

2. Find the z –discontinuity

3. Look for vertical and

forward oriented regions

4. Check the collineraity

5. Use geometrical laws

in order to find

x, y, z and b.

ToF

ToF

IFM O3D2xx.

1. Remove weak defined

pixels.

2. Find the z –discontinuity

3. Look for vertical and

forward oriented regions

4. Check the collineraity

5. Use geometrical laws

in order to find

x, y, z and b.

% Program: 'CrossRatio_VR_diagonal_O2.m'.

3/5

Time of Flight cameras: vector interpretation

Reference =

TOF-source Measured

I0

I0 = b + A.ei.(φd + θ)

I1 = b + A.ei.(φd + θ + π/2)

I2 = b + A.ei.(φd + θ + π)

I3 = b + A.ei.(φd + θ + 3π/2)

b = (I0 + I1 + I2 + I3 )/4

A = (I0 + I1.e-iπ/2 + I2.e

-i.π + I3.e

-i.3π/2 )/4

φd + θ = arg(A)

I1

I2

I3

The calculation can easely be extended for:

- non-harmonic periodic illumination ( Fourier expansion !) ,

- more than 4 equidistant measurings in one cycle .

θ

φd

Remember: vector r.eiφ = r.[ cos(φ) + i.sin(φ) ]

r.cos(θ)

r.sin(θ)

b