Technical ArchitectureAntwerp IoT platform
The architecture should be based on the specific requirements of this project. If requirements are unclear, we have to take our best bet and clarify our assumptions.
The platform should be generic enough to handle many use cases. Each use case should be specific enough to address a concrete city-related challenge.
In the spirit of Lean, we should expect to make continuous changes to our architecture, as assumptions are validated and new requirements come in.
Objective: define the technical components needed for an IoT architecture which fit in the functional design
We have gathered some valuable requirements,although we don’t know all use cases up-front
Data consumption- Easy data querying and data exploration- Open Data API- Easy-to-use data visualisations
3rd party integrations- Allow 3rd parties to add sensors, use data,
and build apps- Integrate with rest of AcPaaS
Security & Privacy- Apply security best practices- Guarantee citizen privacy- Some authorised groups can have
elevated access
IoT devices- many different devices, technologies, data
formats, protocols- Collect data over different networks- Control sensors
Big Data- High Volume, Velocity and Variety- Build a Data Lake to allow reuse of data
Analytics- Streaming, (near) Real-Time Analytics- Complex Batch Analytics- BI reporting
How do we send commands to an IoT device? How do we control IoT devices?
Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?
Aspects of an IoT platform
IoTCommands
3rd partyExtensions
Data Infrastructure
DataPipeline
How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?
From data ingestion to data consumption. How is the data stored, processed and queried?
How do we send commands to an IoT device? How do we control IoT devices?
Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?
Aspects of an IoT platform
IoTCommands
3rd partyExtensions
Data Infrastructure
DataPipeline
How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?
From data ingestion to data consumption. How is the data stored, processed and queried?
Evaluate cloud experience POC and make a plan for production deployments
Find on-premise alternatives for Cloud Services
Make a TCO analysis comparing costs for cloud versus on-premise
Evaluate scalability constraints
Proposal: Go to the cloud for the POCs.Make a final decision for production deployments.
Cloud is more suited for POCs than on-premise
IoT is a new field and we’ll have to adapt to new hardware, technologies, communication protocols and business use cases.
We are in POC mode. We don’t have a clear business case yet. Therefore we want to limit investments both in terms of hardware and in terms of development costs.
Cloud offers more flexibility, lower hardware investments and lower development costs, compared to on-premise.
How do we send commands to an IoT device? How do we control IoT devices?
Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?
Aspects of an IoT platform
IoTCommands
3rd partyExtensions
Data Infrastructure
DataPipeline
How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?
From data ingestion to data consumption. How is the data stored, processed and queried?
The 5 phases of a data analytics pipeline
Ingestion Storage Processing Querying Consumption
Ingestion collects data from sensors and reference sources, and uploads it to the platform.
Data is stored on arrival to allow for analytics later and to serve as an archive.
Processing converts the raw data to a uniform format and joins data together for complex analytics.
The data is loaded into a database for querying and analysis.
The data is being consumed by external tools and programs to deliver insights.
Our design will be based on a modular architecture. This allows for the following properties:
- Flexibility: Components can be swapped independently - Scalability: Each component can be scaled individually to handle peak loads- Simplicity: Each phase only does one thing, which makes it easy to understand
There are plenty of technologies at each phase. The choice depends on the requirements.
ApacheNifi
ApacheFlume
ApacheSqoop
Kafka Connect
MashapeKong
Logstash
AmazonS3
ApacheCassandra
ApacheKafka
SANStorage
ApacheHDFS
WSO2ESB
AWSEMR
ApacheSpark
ApacheFlink
AWSLambda
Python
Pentaho
ElasticSearch
MongoDB
AWSRedshift
HPVertica
Postgres
ApacheCassandra
Kibana
D3.js
IBMCognos
DataAPI
Datylon
Tableau
Ingestion ConsumptionStorage Processing Querying
An example data pipeline could look like this:
Ingestion
Data API
ConsumptionStorage Processing Querying
S3
EMR
How this design fits in the functional architecturedesigned by Continuum
?
How this design fits in the functional architecturedesigned by Continuum
Ow
n se
nsor
s3r
d pa
rty
sens
ors
3rd
party
se
nsor
s
Ow
n co
mm
unic
atio
n br
idge
AP
I3r
d pa
rty
Physical Devices
Access control & Transformation Data Storage Data
consumption
Access control & transformation
Data Exposure
AP
I Gatew
ay
Security
Trusted3rd parties
Internet P
ublic Dom
ain
Sec
urity
How do we send commands to an IoT device? How do we control IoT devices?
Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?
Aspects of an IoT platform
IoTCommands
3rd partyExtensions
Data Infrastructure
DataPipeline
How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?
From data ingestion to data consumption. How is the data stored, processed and queried?
3rd parties can be involved as data providers, data consumers and app builders
Data providers
Data consumers
AppBuilders
How do we send commands to an IoT device? How do we control IoT devices?
Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?
Aspects of an IoT platform
IoTCommands
3rd partyExtensions
Data Infrastructure
DataPipeline
How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?
From data ingestion to data consumption. How is the data stored, processed and queried?
IoT commands pass through the same security gateways O
wn
sens
ors
3rd
party
se
nsor
s3r
d pa
rty
sens
ors
Ow
n co
mm
unic
atio
n br
idge
AP
I3r
d pa
rty
Security
Trusted3rd parties
Internet P
ublic Dom
ain
Command API
AP
I Gatew
ay
How do we send commands to an IoT device? How do we control IoT devices?
Do we go to the cloud or do we stay on-premise?Which cloud provider do we select?
Aspects of an IoT platform
IoTCommands
3rd partyExtensions
Data Infrastructure
DataPipeline
How can we build algorithms for many different use cases?How can we expose these outputs for consumption?How do we allow 3rd parties to add new sensors to the network?
From data ingestion to data consumption. How is the data stored, processed and queried?
Top Related