2007-10-16

MonetDB spatial

1

Onderzoeksinstituut OTB

MonetDB spatial

Een open-source Spatial DBMS

Media Plaza/Het gebruik van OS software in GEO-ICT

Wilko Quak

2007-10-16 2MonetDB spatial

Onderzoeksinstituut OTB

Overzicht

• TUDelft / OTB / Sectie GISt / GeoInfoNed Project• De spatial DBMS markt.• Categorie-en van DBMS gebruikers• Intermezzo: spatial DBMS.• MonetDB en MonetDB (spatial)• De toekomst van MonetDB (spatial)

2007-10-16 3MonetDB spatial

Onderzoeksinstituut OTB

GIS technology

2007-10-16 4MonetDB spatial

Onderzoeksinstituut OTB

GISt visie

Spatial Information Infrastructure (SII), a vision…• Heading towards an information society with

varying impact on sectors of our economy• Impact on the geo-information sector is huge: geo-

info and processing can be transported via (wireless) networks and can be combined with positioning

• Engines of the SII nodes: geo-DBMSs (Data Base Management Systems) filled with geo-data

• Fair pricing of data and services, no more endless copying of data (and loss of quality/up-to-dateness)

2007-10-16 5MonetDB spatial

Onderzoeksinstituut OTB

GISt: scientific/technological goal

• Central research theme geo-DBMS as ‘glue’ between:

• 3D spatio-temporal modeling• Computational geometry• Distributed GI processing• Mobile GIS (LBS)• Knowledge engineering

Ambition: top 1 (3) of geo-DBMS Universities in the world

2007-10-16 6MonetDB spatial

Onderzoeksinstituut OTB

Geo Database Management Center• TU Delft open R&D center ‘geo-DBMS’• Partnerships with ICT industry: Oracle, ESRI,

Bentley• ‘Database lab’ and ‘3D GIS/VR lab’• Website http://www.gdmc.nl for

• Communication/documentation

• Internet GIS

2007-10-16 7MonetDB spatial

Onderzoeksinstituut OTB

Met welk onderzoek kan de Sectie GISt bijdragen?

• Onderzoek naar bruikbaarheid bestaande producten (benchmarken)

• Wat mist er nog• Meedenken met ontwikkelingen• Eigen DBMS ontwikkelen als research vehikel:

GeoInfoNed Project

2007-10-16 8MonetDB spatial

Onderzoeksinstituut OTB

GeoInfoNed – RGI-232

2007-10-16 9MonetDB spatial

Onderzoeksinstituut OTB

wie

• CWI – Leading DBMS experts with MonetDB

• TUDelft/OTB – Knowledge of spatial processes

• CycloMedia – Huge dataset and interesting problems

• RWS/AGI – Large and diverse datasets and interesting problems

2007-10-16 10MonetDB spatial

Onderzoeksinstituut OTB

Wat

• Is er nog wel ruimte op de markt? Een inventarisatie:• Aanbieders• Gebruikers

We gaan een spatial DBMS bouwen en daar onderzoek mee doen.

2007-10-16 11MonetDB spatial

Onderzoeksinstituut OTB

De spatial DBMS markt (aanbieders)

• Oracle – Marktleider commerciele segment• Informix/DB2 – Prima implemenatie. Niet erg

actief• PostgreSQL/PostGIS – ‘Marktleider’ open-source• SQLServer – komt binnenkort• MySQL – open-source, beperkte functionaliteit

2007-10-16 12MonetDB spatial

Onderzoeksinstituut OTB

Spatial DBMS markt (gebuikers)

4 classes:1. Server Builders: publish spatial data via web

server2. GIS User: Load various datasets and perform

complex analyses3. Data Maintainer: Maintain one core dataset4. Power Users: All of the above and more

2007-10-16 13MonetDB spatial

Onderzoeksinstituut OTB

Class 1: Web Server Builders

• Runs under GeoServer• You do not really need a DBMS for this (You

use a fraction of DBMS functionality)• Only one query counts: Find everything within

BBOX.• Multi-User• Small Queries

2007-10-16 14MonetDB spatial

Onderzoeksinstituut OTB

Class 2: GIS users

• Full fledges GIS is much more powerful but SQL experts might be tempted

• Main interest is functionality• Spend more time on loading data• Need a good query optimizer• Data mining

2007-10-16 15MonetDB spatial

Onderzoeksinstituut OTB

Class 3: Dataset Maintainers

• Limited number of queries.• 24/7 stability• Transactions.• Clustering of data after updates is interesting.

2007-10-16 16MonetDB spatial

Onderzoeksinstituut OTB

Class 4: Power users

• All of the above and more• Take time to understand software• Run into research problems:

• Point clouds from laser-scanning• Support 3d-volumes• Realtime dynamic data

2007-10-16 17MonetDB spatial

Onderzoeksinstituut OTB

Intermezzo: Spatial DBMS

• Spatial data types• Operaties op types• Indexeren op types

2007-10-16 18MonetDB spatial

Onderzoeksinstituut OTB

Spatial Data Types

• Basic types: Point Line Polygon

• More Complex: 3D, Geodetic Coordinates

2007-10-16 19MonetDB spatial

Onderzoeksinstituut OTB

Spatial Data types volgens OpenGIS

PointLineStringLinearRingPolygonMultiPointMultiLineStringMultiPolygon

2007-10-16 20MonetDB spatial

Onderzoeksinstituut OTB

Functions op spatial types

2007-10-16 21MonetDB spatial

Onderzoeksinstituut OTB

• R-tree puts MBR on object- MBR : minimum bounding rectangle- Object can be of type point, line or region- At lowest (leaf) level objects are grouped

to larger MBR’s, which can be grouped (and so on .....)

Indexeren spatial types

D

F

E

G

H

J

K

A

C

L

N

MB

I

A B C

D E F G H I J K L M N

Branching factor 4

Einde Intermezzo

2007-10-16 22MonetDB spatial

Onderzoeksinstituut OTB

MonetDB Introduction*

MonetDB is opens• Hardware trends• MonetDB design considerations• MonetDB architecture

*Slides borrowed from CWI

2007-10-16 MonetDB spatial

Onderzoeksinstituut OTB

Hardware Trends50% p/year:

- cpu speed

- mem size

- mem bandwidth

- disk bandwidth1% p/year:

- mem latency

10% p/year:

- disk latency

2007-10-16 MonetDB spatial

Onderzoeksinstituut OTB

Latency is the enemy!

• Commercial DBMS products (oracle, DB2, SQLserver) stem from OLTP roots

• focus on minimizing random I/Os => depend on latency!

• MonetDB: built for bulk access• optimize CPU and memory performance

2007-10-16 25MonetDB spatial

Onderzoeksinstituut OTB

MonetDB design considerations

• Multi-model database kernel support• Extensible data types, operators, accelerators• Database hot-set is memory resident• Simple data structures are better• Index management should be automatic• Do not replicate the operating system• Optimize when you know the situation• Cooperative transaction management

2007-10-16 26MonetDB spatial

Onderzoeksinstituut OTB

Monetkernels

MAPI protocol

JDBC

C-mapi lib

Perl

End-user application

ODBC PHP Python

SQL XQuery

MonetDB product family

2007-10-16 27MonetDB spatial

Onderzoeksinstituut OTB

SWOT

State-of-the-art SQL & XQuery DBMS; multi-model database support; decomposed storage model; query processing for decision support systems; extensible kernel; multiple APIs;…..

strengthTriggers and SQL procs missing; XQuery; some overhead in simple SQL queries; MIL scripting; missing GIS module

weakness

Commercial systems requirements; competition established market; floating interest; development skills and quality awareness

threats

Unique selling points for advanced applications in multimedia, streaming, covering both small and the large world

opportunities

GET INSPIRED BY ADVANCED APPLICATIONS

2007-10-16 28MonetDB spatial

Onderzoeksinstituut OTB

MonetDB - Physical data organization• Binary Association Tables

ID Day Discount10 4/4/98 0.19511 9/4/98 0.06512 1/2/98 0.17513 7/2/98 0

OID ID100 10101 11102 12103 13104 14

OID Day100 4/4/98101 9/4/98102 1/2/98103 7/2/98104 1/2/99

OID Discount100 0.195101 0.065102 0.175103 0104 0.065

2007-10-16 29MonetDB spatial

Onderzoeksinstituut OTB

Spatial MonetDB

2007-10-16 30MonetDB spatial

Onderzoeksinstituut OTB

Spatial MonetDB -- nu

• Eerste implementatie in volgende release (Q4).• OpenGIS implementatie gebaseerd op PostGIS

• OpenJUMP plugin• Shape2MonetDB converter • GeoTools Plugin (in ontwikkeling)

2007-10-16 31MonetDB spatial

Onderzoeksinstituut OTB

Spatial MonetDB – de toekomst

• MonetDB moet het hebben van zijn speciale architectuur.

• Uitdagingen:• Puntenwolken van Laserscanner.• Data-minen op grote spatio-temporele

datasets.• XML-Data.

• Saaier werk:• Testen huidige implementatie• debuggen

2007-10-16 32MonetDB spatial

Onderzoeksinstituut OTB

Vragen

Discussie