Technical ArchitectureAntwerp IoT platform

The architecture should be based on the specific requirements of this project. If requirements are unclear, we have to take our best bet and clarify our assumptions.

The platform should be generic enough to handle many use cases. Each use case should be specific enough to address a concrete city-related challenge.

In the spirit of Lean, we should expect to make continuous changes to our architecture, as assumptions are validated and new requirements come in.

Objective: define the technical components needed for an IoT architecture which fit in the functional design

We have gathered some valuable requirements,although we don’t know all use cases up-front

Data consumption- Easy data querying and data exploration- Open Data API- Easy-to-use data visualisations

3rd party integrations- Allow 3rd parties to add sensors, use data,

and build apps- Integrate with rest of AcPaaS

Security & Privacy- Apply security best practices- Guarantee citizen privacy- Some authorised groups can have

elevated access

IoT devices- many different devices, technologies, data

formats, protocols- Collect data over different networks- Control sensors

Big Data- High Volume, Velocity and Variety- Build a Data Lake to allow reuse of data

Analytics- Streaming, (near) Real-Time Analytics- Complex Batch Analytics- BI reporting

How do we send commands to an IoT device? How do we control IoT devices?

Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?

Aspects of an IoT platform

IoTCommands

3rd partyExtensions

Data Infrastructure

DataPipeline

How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?

From data ingestion to data consumption. How is the data stored, processed and queried?

How do we send commands to an IoT device? How do we control IoT devices?

Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?

Aspects of an IoT platform

IoTCommands

3rd partyExtensions

Data Infrastructure

DataPipeline

How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?

From data ingestion to data consumption. How is the data stored, processed and queried?

Evaluate cloud experience POC and make a plan for production deployments

Find on-premise alternatives for Cloud Services

Make a TCO analysis comparing costs for cloud versus on-premise

Evaluate scalability constraints

Proposal: Go to the cloud for the POCs.Make a final decision for production deployments.

Cloud is more suited for POCs than on-premise

IoT is a new field and we’ll have to adapt to new hardware, technologies, communication protocols and business use cases.

We are in POC mode. We don’t have a clear business case yet. Therefore we want to limit investments both in terms of hardware and in terms of development costs.

Cloud offers more flexibility, lower hardware investments and lower development costs, compared to on-premise.

How do we send commands to an IoT device? How do we control IoT devices?

Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?

Aspects of an IoT platform

IoTCommands

3rd partyExtensions

Data Infrastructure

DataPipeline

How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?

From data ingestion to data consumption. How is the data stored, processed and queried?

The 5 phases of a data analytics pipeline

Ingestion Storage Processing Querying Consumption

Ingestion collects data from sensors and reference sources, and uploads it to the platform.

Data is stored on arrival to allow for analytics later and to serve as an archive.

Processing converts the raw data to a uniform format and joins data together for complex analytics.

The data is loaded into a database for querying and analysis.

The data is being consumed by external tools and programs to deliver insights.

Our design will be based on a modular architecture. This allows for the following properties:

- Flexibility: Components can be swapped independently - Scalability: Each component can be scaled individually to handle peak loads- Simplicity: Each phase only does one thing, which makes it easy to understand

There are plenty of technologies at each phase. The choice depends on the requirements.

ApacheNifi

ApacheFlume

ApacheSqoop

Kafka Connect

MashapeKong

Logstash

AmazonS3

ApacheCassandra

ApacheKafka

SANStorage

ApacheHDFS

WSO2ESB

AWSEMR

ApacheSpark

ApacheFlink

AWSLambda

Python

Pentaho

ElasticSearch

MongoDB

AWSRedshift

HPVertica

Postgres

ApacheCassandra

Kibana

D3.js

IBMCognos

DataAPI

Datylon

Tableau

Ingestion ConsumptionStorage Processing Querying

An example data pipeline could look like this:

Ingestion

Data API

ConsumptionStorage Processing Querying

S3

EMR

How this design fits in the functional architecturedesigned by Continuum

?

How this design fits in the functional architecturedesigned by Continuum

Ow

n se

nsor

s3r

d pa

rty

sens

ors

3rd

party

se

nsor

s

Ow

n co

mm

unic

atio

n br

idge

AP

I3r

d pa

rty

Physical Devices

Access control & Transformation Data Storage Data

consumption

Access control & transformation

Data Exposure

AP

I Gatew

ay

Security

Trusted3rd parties

Internet P

ublic Dom

ain

Sec

urity

How do we send commands to an IoT device? How do we control IoT devices?

Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?

Aspects of an IoT platform

IoTCommands

3rd partyExtensions

Data Infrastructure

DataPipeline

How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?

From data ingestion to data consumption. How is the data stored, processed and queried?

3rd parties can be involved as data providers, data consumers and app builders

Data providers

Data consumers

AppBuilders

How do we send commands to an IoT device? How do we control IoT devices?

Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?

Aspects of an IoT platform

IoTCommands

3rd partyExtensions

Data Infrastructure

DataPipeline

How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?

From data ingestion to data consumption. How is the data stored, processed and queried?

IoT commands pass through the same security gateways O

wn

sens

ors

3rd

party

se

nsor

s3r

d pa

rty

sens

ors

Ow

n co

mm

unic

atio

n br

idge

AP

I3r

d pa

rty

Security

Trusted3rd parties

Internet P

ublic Dom

ain

Command API

AP

I Gatew

ay

How do we send commands to an IoT device? How do we control IoT devices?

Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?

Aspects of an IoT platform

IoTCommands

3rd partyExtensions

Data Infrastructure

DataPipeline

How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?

From data ingestion to data consumption. How is the data stored, processed and queried?