Building a Realtime Streaming System for a Smart City

Name: Building a Realtime Streaming System for a Smart City
Rating: 3.6 (3 reviews)

Master Real-Time Data Engineering: Build a Smart City End to End Streaming Pipeline with Kafka, Spark, AWS & Docker

Created byGaniyu Yusuf

Last updated 1/2026

English

What you'll learn

Participants will gain a thorough understanding of data engineering concepts, including data ingestion, processing, storage, and visualization.
Through practical demonstrations and coding exercises, participants will gain hands-on experience with a variety of industry-standard tools and technologies
By following along with the project setup, coding, and deployment process, participants will acquire the skills needed to build and deploy real-world solutions
Participants will encounter challenges and learn how to troubleshoot common issues that arise during the development and deployment of a data streaming pipeline
Participants will be well-equipped to pursue career opportunities in data engineering, IoT, cloud computing, and related fields.

Course content

5 sections • 23 lectures • 2h 27m total length

Project Background and Use case1:08
Learn real-time streaming with Apache Spark by building an IoT vehicle data pipeline that collects GPS, weather, medical, and accident data for live analysis from London to Birmingham.
Setting up Docker on your Computer2:10
Install and configure Docker on your computer, choosing the right Docker Desktop version for Mac (Apple Silicon or Intel), and verify containers and images while preparing for Kubernetes.

System architecture4:23
Explore the system architecture for streaming vehicle data—from GPS and camera to emergencies—through Kafka, Zookeeper, and Spark, into AWS S3, Glue, IAM, and Redshift for analytics.
Creating the Project2:44
Setting up zookeeper and apache Kafka on docker9:00
Set up zookeeper and apache kafka on docker with a docker-compose file, using confluent zookeeper and broker, health checks, and a data mastery lab network.
Setting up Apache Spark master worker architecture7:41
Configure an Apache Spark master–worker architecture with Bitnami Spark images and a shared Spark common module, mapping volumes for jobs and exposing 7077 and 1990 ports for realtime streaming.

Writing the base code for IOT data production10:53
Build the base IoT data producer in main.py and spark city, install confluent kafka and PySpark, configure vehicle, gps, weather, and emergency topics, and use environment variables for setup.
Generating vehicle location10:39
Generating GPS data2:43
Generating Camera Information3:01
Generate traffic camera data by capturing a device id, camera id, timestamp, and a uuid v4, then attach a base64 snapshot or a camera URL stored in an S3 bucket.
Generating Weather data5:36
Generate weather data for a smart city streaming system using device id, timestamp, and location, with deterministic attributes like temperature, wind, humidity, and air quality index via seeded random generator.
Generating emergency incidence data4:04
Generate emergency incident data with device IDs, timestamps, and locations, create unique incident IDs and types, and publish real-time streams to Kafka for smart city monitoring.
Producing IOT data to apache kafka11:31
Learn to produce IoT data to Apache Kafka by building a reusable producer, serializing to JSON, and validating delivery across vehicle, GPS, traffic, weather, and emergency topics.
Setting up and AWS secret key and access key4:53
Set up AWS credentials and S3 bucket permissions for streaming data, generate access key and secret key via IAM, and configure your app to use them securely.

Setting up apache spark consumer from kafka10:22
Creating schema for the spark streams7:08
Develop vehicle, gps, traffic, weather, and emergency schemas as spark sql structs, including id, device id, timestamp, and location, then create a reusable reader to avoid repetition.
Connecting schema to spark streams6:42
Build a Spark streaming pipeline that reads Kafka topics into vehicle, GPS, traffic, weather, and emergency, parses JSON with the schema, and applies a timestamp watermark before storing to S3.
Streaming data to s3 lakehouse16:38

Creating AWS Glue crawlers for s310:32
Learn how to use AWS Glue crawlers to scan S3 data, build a data catalog, and connect to Athena for real-time smart city streams, producing five tables.
Querying data with AWS Athena2:47
Creating Redshift cluster4:52
Create a large single-node Amazon Redshift cluster for a smart city, attach an S3 read-only IAM role, and retrieve the endpoint and JDBC/ODBC URLs for connection.
Loading and querying Redshift8:06
Connect to Redshift and load data from the Glue data catalog, create an external schema named smart city, and query the vehicle gps data with appropriate iam permissions.
Course Resources0:01

Requirements

Basic Python programming experience

Description

In this comprehensive Udemy course, you'll be building a sophisticated Smart City End to End Realtime data streaming pipeline, covering every aspect from data ingestion to processing, and finally storage and visualization.

Throughout this hands-on course, you'll leverage an arsenal of industry-leading tools and technologies, including Apache Kafka for high-throughput, fault-tolerant messaging, Apache Spark for real-time data processing, Docker for containerization, and a suite of AWS services such as S3, Glue, Athena, IAM, and Redshift for cloud-based data storage, management, and analytics.

What's in the course?

Architect a Smart City End to End Realtime data streaming pipeline
Set up and configure Docker containers for development and deployment
Code IoT service producers for generating diverse data streams including vehicle information, GPS coordinates, traffic updates, weather conditions, and emergency incidents
Stream data into Apache Kafka for real-time processing and distribution
Utilize AWS services including S3, Glue, Athena, IAM, and Redshift for cloud-based data storage, management, and analytics
Configure S3 buckets with policies and manage IAM roles and credentials for secure access
Use AWS Glue for data cataloging and transformation, enabling seamless integration with downstream analytics tools
Query data stored in AWS S3 with Athena, executing powerful SQL queries on your data lake
Set up and manage a scalable data warehouse with AWS Redshift for advanced analytics and visualization
Troubleshoot, debug, and optimize your streaming solution for maximum performance and reliability
Build a portfolio project showcasing your proficiency in real-time data engineering and cloud-based analytics

But that's just the beginning! You'll delve into the intricacies of AWS setup, learning how to configure S3 buckets with policies, manage IAM roles and credentials, and leverage AWS Glue for data cataloging and transformation. With Athena, you'll execute powerful queries on your data lake, while Redshift will serve as your data warehouse for scalable analytics.

By the end of this course, you'll not only have built a fully functional Smart City data pipeline, but you'll also have polished your skills in troubleshooting, debugging, and optimizing your streaming solution.

Enroll now and unlock the door to endless possibilities in the realm of data engineering!

Who this course is for:

Everybody interested in building scalable and efficient systems

Building a Realtime Streaming System for a Smart City

What you'll learn

Explore related topics

Course content

Introduction2 lectures • 3min

System Architecture4 lectures • 24min

IOT Data Producer8 lectures • 53min

Realtime Streaming Consumer4 lectures • 41min

Data Transformation on AWS5 lectures • 26min

Requirements

Description

Who this course is for: