Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Hands-On Training: Streaming Data with Apache Kafka on Azure

Name: Hands-On Training: Streaming Data with Apache Kafka on Azure
Rating: 4.6 (72 reviews)

From Setup to Streaming: Build Real-World Apache Kafka Solutions on Azure

Created byMohammed Al-moayed

Last updated 6/2025

English

What you'll learn

Understand and Set Up Apache Kafka
Implement Kafka Services and Topics
Use Python to Consume Data from Kafka
Use Python to produce Data to Kafka Topic
Apply Apache Kafka in Real-World Scenarios

Course content

11 sections • 39 lectures • 3h 41m total length

Why Learn by Doing?2:01
In this introductory video, I am explains why learning by doing is one of the most effective ways to master new skills. You’ll discover how active engagement enhances understanding and retention, how applying knowledge through hands-on activities strengthens concepts, and how real-world problem-solving builds critical thinking skills.
The video also highlights how experiential learning boosts motivation and makes the learning process more enjoyable and impactful. By the end, you’ll understand why practical, hands-on training is at the heart of this course — preparing you to confidently apply what you learn in real-world Kafka projects.
Top 5 Kafka Use Cases3:14
Explore the top five Kafka use cases—data streaming, log activity tracking, message queuing, change data capture, and data replication.
Brief Introduction to Apache Kafka11:04
Explore Apache Kafka's real-time streaming capabilities for publishing and subscribing to topic messages. Discover core concepts like topics, producers, consumers, brokers, and fault-tolerant, scalable, durable data flow.
Quiz on Introduction

Course Learning Objectives3:11
Prerequisites for This Course1:06
Meet the prerequisites for this course, including basic Python or programming knowledge and a desire to learn. Gain hands-on experience with streaming and understand how Apache Kafka works on Azure.
Overview of the Course Use Case2:25
Explore a two-virtual-machine architecture for streaming data with Apache Kafka on Azure, featuring a Python Flask API, Kafka producers and consumers, and outputs to CSV, data lake, and Databricks.
Quiz on the use Case

Setting Up Your Azure Account6:26
Essential Concepts for New Students in Azure4:03
Explore essential Azure concepts for new students, including on-demand infrastructure, pay-as-you-go pricing, resource groups, and subscriptions, with provisioning of virtual machines and services like Azure SQL database and storage.
Creating a Resource Group in Azure2:34
Create an Azure resource group, a container for all related resources, then name and region (Germany) to support proximity while enabling streaming project connections between sources and destinations.
Creating a Virtual Machine for Kafka Server5:40
Create an Ubuntu server virtual machine on Azure to install and run Apache Kafka, leveraging Ubuntu's stability, security, long-term support, and open source cost-efficiency.
Testing Connection to Kafka Server VM2:13
Creating a Virtual Machine for Flask API Server4:29
Testing Connection to Flask API Server VM1:36
Quiz on Hands-On Foundations

Setting Up Zookeeper Service8:35
Setting Up Kafka Service4:28
Starting Zookeeper and Kafka Services3:08
Start Zookeeper and Kafka services with systemctl commands, verify their active status, and prepare to create topics in Kafka.
Creating a Kafka Topic10:21
Best Practices for Topic Creation3:17
Testing Producers and Consumers with Command-Line Tools6:53
Test producing and consuming data with Kafka console producer and consumer commands using two Putty windows, starting Zookeeper and Kafka, and validating topic messages end to end.

Scenario 3: Creating Azure Databricks Workspace10:41
Create an Azure Databricks workspace, configure a cluster, and consume data from a Kafka topic to store in a data lake table, enabling real-time analytics on streaming data.
Scenario 3: Creating Databricks Compute Cluster4:02
Create a databricks compute cluster in the azure databricks workspace, choose a single node to save costs with the latest runtime, and import notebooks to consume kafka topics.
Scenario 3: Consuming Data Using Databricks Application10:20
Consume data from Azure via a Databricks application, streaming from Kafka topic to a Delta table, then read from Delta Lake with Spark and SQL dashboards.
Scenario 3: Registering & Optimizing Delta Table10:09
Scenario 3: Versioning in Delta Table7:14

Requirements

Basic Python or programming knowledge is needed, along with basic familiarity with Azure and Databricks, and a basic understanding of Unix.

Description

Most Apache Kafka courses explain concepts but never show how to build a complete real-world streaming system.

In this course, you will build a real-time data streaming pipeline from scratch using Apache Kafka on Microsoft Azure. Through practical, hands-on exercises, you will learn how to install, configure, and run Kafka in a cloud environment while implementing real data streaming scenarios.

This course focuses on learning by doing. Instead of only studying theory, you will deploy Kafka on Azure virtual machines, produce and consume streaming data using Python, and integrate your pipeline with Azure services such as Azure Data Lake and Azure Databricks.

By the end of the course, you will understand how real-time streaming architectures are built and how Apache Kafka is used in modern data engineering platforms.

What You Will Learn

Apache Kafka Fundamentals
Understand the architecture of Kafka, including brokers, topics, partitions, producers, and consumers.

Deploying Kafka on Azure
Learn how to create and configure Azure infrastructure to run Kafka clusters on virtual machines.

Hands-On Kafka Installation and Configuration
Install and configure Kafka and Zookeeper step-by-step and understand how they work together.

Building Real-Time Streaming Pipelines
Create streaming data workflows using Python producers and consumers.

Integration with Azure Data Platform
Stream data to Azure Data Lake and process it using Azure Databricks and Delta Tables.

End-to-End Streaming Project
Implement and run multiple real-world scenarios demonstrating how Kafka supports scalable real-time data processing.

Why Take This Course?

This course is designed around practical implementation, not just theoretical explanations.

You will:

• Deploy Apache Kafka in a real cloud environment
• Build a complete streaming pipeline from producer to analytics
• Work with industry tools such as Python, Azure Databricks, Delta Tables, and Data Lakes
• Understand how streaming architectures are implemented in real-world projects

By the end of the course, you will have built and tested a full streaming data architecture using Apache Kafka on Azure.

Who This Course Is For

This course is ideal for:

• Aspiring Data Engineers who want hands-on experience with streaming technologies
• Developers interested in real-time data processing and event-driven architectures
• Cloud engineers working with Azure and modern data platforms
• Anyone who wants to understand how Apache Kafka works in real-world data engineering scenarios

Who this course is for:

Have basic knowledge of Python or programming in general.
Are interested in learning hands-on about Apache Kafka, focusing on real-world applications.
Have some basics familiarity with Azure, Databricks, and a basic understanding of Unix.
Are looking to gain practical skills in streaming data and understand how Apache Kafka works in real-world business cases.

Hands-On Training: Streaming Data with Apache Kafka on Azure

What you'll learn

Explore related topics

Course content

Course Kickoff3 lectures • 16min

Course Preparation and Expectations3 lectures • 7min

Hands-On Foundations: Part 17 lectures • 27min

Hands-On Foundations: Part 24 lectures • 24min

Hands-On Foundations: Part 34 lectures • 13min

Hands-On Foundations: Part 46 lectures • 37min

Python Data Consumption2 lectures • 18min

Implementing Use Case Scenarios: Part 13 lectures • 28min

Implementing Use Case Scenarios-part 25 lectures • 42min

Running all scenarios1 lecture • 7min

Requirements

Description

Who this course is for: