
Engage in a hands-on IoT project using Kafka and GridDB to enable near real-time and batch data processing within a Lambda-architecture approach.
Explore the IoT lambda architecture and containerized infrastructure, learning how Docker and Docker Compose orchestrate nine services, including GridDB, Kafka, and Python containers, to build and run the project.
Build and run the IoT project with Docker Compose, managing Grid DB, Kafka, and Zookeeper, verify connectivity, and deploy the Grid DB Sync using Kafka Connect.
GridDB provides persistent, time series–optimized storage for IoT data. Its in-memory architecture supports real-time and batch processing with a programmable Python/Java interface and optional SQL access.
Set up and verify a GridDB server in a Docker environment, access the GridDB CLI via SSH, and run TCL and SQL queries to explore time series data.
Explore Kafka, an open source distributed event streaming platform, and how data flows from the gateway simulator through Kafka to Grid DB via Zookeeper, Broker, Schema Registry, and Connect.
Demonstrate running and inspecting Kafka containers—zookeeper, broker, connect, and schema registry—using docker and logs. Show connecting topics to GridDB via a Kafka connector and an automation script.
Explore how the gateway simulator, a Python script, generates simulated utility meter data with timestamped readings and streams it via Kafka to GridDB.
Build and run the gateway simulator with docker compose, deploy gateway sim and Kafka producer to simulate 10 devices, push data to the broker, and verify persistence in GridDB.
Develop a batch bill generation process in Java for GridDB, computing monthly meters' total kilowatt hours and daily usage via multi queries, with cron or docker-based automation.
Execute the bill jobs to generate meter and sensor reports in GridDB, verifying data and Kafka topics with the GridDB CLI and Docker Compose.
Explore the grandstack architecture - grid db, express.js, react, and node.js - and how a full stack API delivers IoT data from grid db to a reactive front end.
Verify grid DB connectivity, run Kafka and build jobs, then visualize meter data and billing history by querying date range with NodeJS and React, and render charts in the browser.
Analyze IoT data using Jupyter notebook connected to GridDB server, run in your browser via a dockerized environment, using NumPy and pandas to run queries like select all.
Operate in a browser-based Jupyter notebook to query GridDB, build pandas data frames, and visualize metrics with matplotlib using numpy for stats.
This course aims to teach you about building an IoT platform with a real-life, hands-on project!
We will begin by utilizing GridDB to create an IoT project which will simulate data from IoT Sensors, stream that data through Kafka, sent to GridDB, and then utilized in a variety of ways, including a real-time data visualizer. The data visualizer will be made up from node.js.
In the first parts of the course, we will delve into what Docker is and how it ties into this project; in our case, we have containerized all different aspects of the project to be able to easily share everything, with the added benefit of teaching more about using docker in conjunction with Kafka, GridDB, Jupyter Notebook, etc.
Next, we will talk about Kafka (outside of the context of Docker), what it is, what it does, and how it ties into ours and other IoT projects' workflows. In our case, Kafka handles data streaming from sensors to GridDB, adding reliability, stability, and robustness. From there, we talk about how to build a small python script to be simulate an IoT gateway, which teaches how to send data from somewhere, through Kafka and finally through to GridDB.
After that, we go through and discuss using java to code an bill report-generation application which pulls all data from our database. Next up is data visualization: we will introduce the GERN stack and show you how to query data from GridDB using nodejs to create real-time data charts.
And lastly we will showcase using Python and jupyter to do some data analysis of your GridDB data.