Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x •...

51
Building a self-driving RC car [email protected] Bert Jan Schrijver @bjschrijver

Transcript of Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x •...

Page 1: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

Building a self-driving RC car

[email protected] Jan Schrijver

@bjschrijver

Page 2: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 3: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@bjschrijver@TimvEijndhoven @bjschrijver

Let’s meet

Bert Jan Schrijver

Page 4: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@bjschrijver@TimvEijndhoven @bjschrijver

Let’s meet

Tim van Eijndhoven

Page 5: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

How it all started…

Page 6: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

RoboRace challenge

Page 7: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

The rules• 4 teams • Each team gets:

• RC car kit • Fixed budget (150 euro)

• Three races: 1. drag race (start, drive, stop) 2. race track (race 1 + corners) 3. destruction derby ;-)

(race 2 with multiple cars on same track)

Page 8: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

The kit

Page 9: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

The brain

Page 10: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

The platform

Page 11: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Vert.x• Toolkit for building reactive applications on the JVM • Event-driven, non-blocking • General purpose application framework • Why Vert.x for our project? • Lightweight, fast • Polyglot • Distributed eventbus (browser included!)

• Building microservices with Vert.x: https://youtu.be/yLg-LPSRjho

Page 12: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Connecting the dots

Page 13: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

• General purpose input/output

GPIO

Page 14: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Speaking the same language

Page 15: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

PWM• Pulse-width modulation, repeating pulses of variable width • ServoBlaster: https://github.com/richardghirst/PiBits/tree/

master/ServoBlaster

Page 16: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Staying safe

Page 17: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Getting visual

Page 18: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

RPi Cam Web Interface• Web interface for the Raspberry Pi Camera module • Video streaming • Web interface to configure video settings https://github.com/silvanmelchior/RPi_Cam_Web_Interface

Page 19: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Ready for the road

Page 20: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Putting stuff together

Page 21: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Learning to drive…

Page 22: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

OpenCV• Open source computer vision and machine learning library

written in C++ with Java interfaces • Optimised algorithms for computer vision • Most widely used computer vision library • Many resources available

Page 23: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

1: Canny edge detection 2: Probabilistic Hough Line Transform

3: Filter group and average lines 4: Determine lane center

Page 24: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Show me some code…

Page 25: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Boars Nest (Laptop) Duke Farm (Car)

Daisy

Luke

Bo

Serv

obla

ster

Boss Hogg

Roscoe

Flash Gen

eral

Lee

Heartbeat

Manual inputs

Steering / Speed instructions

Start/Stop autopilot

Image feed

Angle &Distance

IntermediateCV images

Page 26: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Expectation

Page 27: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Reality

Page 28: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 29: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 30: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 31: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 32: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 33: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Let’s race!

Page 34: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 35: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 36: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

In the real world

Page 37: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 38: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 39: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 40: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 41: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Page 42: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Examples of safety measures • Geographic area limitation • Strictly defined conditions • Redundancy • Multiple sensors • Isolating control systems • Human intervention fallback

Page 43: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Weakness of AI for computer vision

Page 44: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Ethics

Page 45: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

http://moralmachine.mit.edu

Page 46: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

http://moralmachine.mit.edu

Page 47: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

“In hazardous situations that prove to be unavoidable, despite all technological precautions being taken, the protection of human life enjoys top priority in a balancing of legally protected interests.”

Page 48: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

“In the event of unavoidable accident situations, any distinction based on personal features (age, gender, physical or mental constitution) is strictly prohibited.”

Page 49: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

@TimvEijndhoven @bjschrijver

Future improvements• Improve detection and navigation of curved lanes • Optimize line detection algorithm using sliding window • Automatic (re)calibration of:

• Video parameters • OpenCV parameters

• Add Flight recorder to enable replay and debug • Control servos directly from Java • Use AI for navigation • Optimize for running everything on the car

Page 50: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

Questions?

@TimvEijndhoven@bjschrijver

Page 51: Building a self-driving RC car - TNG Technology Consulting...@bjschrijver @TimvEijndhoven Vert.x • Toolkit for building reactive applications on the JVM • Event-driven, non-blocking

Thanks for your time.Got feedback? Tweet it!

All pictures belongto their respective

authors

https://github.com/rc-dukes

@TimvEijndhoven@bjschrijver