Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Apache Kafka v3 for Big Data Streaming - Python Hands On

Name: Apache Kafka v3 for Big Data Streaming - Python Hands On
Rating: 4.0 (55 reviews)

Build End-to-End Real-Time Streaming Pipelines : Kafka, Flink, Elasticsearch, Kibana in Python - Big Data Architecture

Created byBig Data Landscape

Last updated 12/2023

English

What you'll learn

Master data ingestion: Efficiently process diverse data streams.
Unleash Kafka's power: Understand core concepts for optimal streaming.
Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer...
Python + Kafka: Build practical skills for producers, consumers, Topics, Partitions, Brokers, and more.
Create your Producers and Consumers in Python to interact with Kafka
Real-world Twitter data: Ingest and scale data seamlessly into HDFS.
Streamline with Kafka and Flink: Craft end-to-end pipelines for real-time analytics.
Real-time Analytics and Visualization: Master the art of analyzing streaming data and visualizing insights using Elasticsearch and Kibana in your Kafka pipeline

Course content

10 sections • 47 lectures • 2h 42m total length

Introduction1:53

Overview of Kafka components2:08
Explore the core Kafka components, including brokers, clusters, Zookeeper coordination, topics, producers, and consumers, to understand how data streams are stored, organized, and consumed.
Quiz
kafka topic1:48
Kafka Partitions2:22
Quiz
Kafka Topic Replication3:49
Learn how Kafka topic replication provides fault tolerance by duplicating topic data across multiple brokers, with a leader and followers staying in sync to ensure availability and durability.
Quiz

Kafka CLI Hands-on | Introduction1:24
Apache Kafka Installation Guide - Getting Started with Kafka | Article1:58
How to Setup 3 nodes Kafka Cluster2:01
Start 3 nodes Kafka Cluster3:35
Create Kafka topic with CLI3:30
Delete Kafka Topic with CLI2:31
Delete a Kafka topic via the command line using the Kafka scripts in the bin directory, specifying bootstrap servers and the topic name; note that deletion is irreversible.
Kafka Console CLI: Producer and Consumer2:54
Linux commands used for 3 nodes Kafka Cluster0:02
This file contains all the commands used in the videos of this section :
Start 3 nodes Kafka and Zookeeper servers
Create Kafka Topic
List Existing Kafka Topics
Delete Kafka Topic
Create Kafka Producer Console
Create Kafka Consumer Console

Kafka Producer and Consumer Hands-on in python | Introduction2:15
Learn to build a Kafka producer and consumer in Python with a step-by-step hands-on approach, leveraging Kafka's real-time distributed streaming for practical data pipelines.
Simple Kafka Producer in Python - Part 1 (coding)5:53
Kafka Producer in Python - Part 2 - ( testing)1:06
Test and run the Python Kafka producer while using the console consumer to receive a new topic stream, demonstrating a real-time data pipeline in Python.
Quiz
Kafka Consumer in python -Part 1 - (coding)4:00
Kafka Consumer in python -Part 2 - (running)2:09
Quiz
Complete Kafka (Producer + Consumer) python code resources0:01

The architecture Design of the hands-on project2:29
Design a big data ingestion pipeline with Twitter data, Kafka, and Python. Build a producer and consumer to filter, analyze sentiment, and store data into a data lake via HDFS.
Streaming Twitter Data with Kafka: Real World Big Data Ingestion Python Project1:46
Real time Data Source: Twitter API from Developer Platform1:36
Extracting Twitter Data Stream from API in python7:04
No Twitter API, Don't worry ! | Create Twitter Data Stream Simulator2:00
Create a Tweets Data Kafka Producer2:57
Create a Tweets Data Kafka Consumer6:22
Install Hadoop (HDFS) - Article Tutorial2:19
Kafka Consumer : Store Data in Hadoop HDFS6:47
Complete Python Code resources (Twitter Producer & Consumer + HDFS Consumer)0:02
The Complete Python Code resources provided include implementations for a Twitter Producer and Consumer, as well as an python Kafka HDFS Consumer. These resources are designed to showcase practical examples of working with Twitter data and Hadoop Distributed File System (HDFS) using Python.

Real Time Streaming Architecture Design: Kafka , Flink, Elasticsearch, & Kibana3:06
Requirement for this project (updated versions 3.+.+)1:57
Apache Flink Introduction | Apache Spark -VS- Flink | PyFlink4:27
Spark vs Flink --- Article1:41
Install Apache Flink -- Article1:25
Introduction to Elasticsearch & Kibana | Article2:09
Configure Flink to consume data from a Kafka topic as a data source11:42
Coding a pyflink code which creates a table environment, specifies a Kafka connector JAR file, defines a source table using a DDL statement, executes the DDL statement to create the source table, retrieves the source table, defines a SQL query to select all columns, executes the query, and prints the result.
Configure Flink to write the processed data to a Elasticsearch sink10:37
Real Time Tweets Word Count with pyFlink and Kafka21:32
Complete Python Code : Streaming pipeline0:02

Requirements

Some understanding of Python Programming
Good to have knowledge about Linux command line
Desire to Master Big Data Streaming
Good to have knowledge on big data processing with Flink or Spark

Description

Unleashing the Power of Apache Kafka and Flink: Cutting-Edge Hands-on Experience with real life case studies

This is the only updated Big Data Streaming Course using Kafka with Flink in python !

(Course newly recorded with Kafka 3.3.1, Flink 1.14.4, ES 7.17.7)

Discover the unrivaled potential of Apache Kafka and the hidden gem of data processing, Flink, in our dynamic course. While Flink may be lesser-known than Spark, it's a powerful tool that surpasses its counterparts in certain aspects.

We'll dive deep into Kafka's core concepts, equipping you with the knowledge to build robust streaming pipelines. But we won't stop there – we'll showcase Flink's prowess as we explore real-time data processing and analytics.

Rest assured, all hands-on exercises are meticulously crafted using the latest versions of Kafka and Flink. Forget about outdated code or compatibility issues – we ensure you're working with cutting-edge tools, ready to conquer the real world.

Although Flink may have a smaller community compared to Spark, this presents a unique opportunity for you to become an early adopter and join the pioneering minds pushing the boundaries of streaming analytics.

We'll guide you step-by-step as you build a complete streaming pipeline that captures live Twitter data, processes it in real-time, and unlocks valuable insights. With our carefully crafted exercises, you'll gain practical experience in ingesting, transforming, and analyzing Twitter data using the latest versions of Kafka and Flink.

Imagine harnessing the pulse of social media to gain actionable insights, all in real-time. From sentiment analysis to trending topics, you'll explore the limitless possibilities of Twitter data analytics.

So, step into the future of stream data processing with Kafka and Flink. Enroll now to gain an edge in the industry, with hands-on expertise on the latest versions of these powerful tools.

Don't miss out on this transformative learning experience – the world of real-time data awaits!

Who this course is for:

Developers who want to learn the Data Ingestion, Apache Kafka , Streaming with Apache Flink
Big Data Architects who want to understand how Apache Kafka fits into their solution architecture
Those desiring to build robust streaming pipelines for real-time analytics
Big Data enthusiasts and software developers looking to expand their skill set
Professionals aiming to stay ahead in the rapidly evolving Big Data landscape
Data analysts and professionals in the field of real-time data processing who want to leverage Kafka and Flink for advanced analytics.
Beginners seeking to enter the world of Kafka and Flink streaming and gain practical hands on skills.
Aspiring data engineers and software developers eager to master Kafka, Flink, Elasticsearch, and Kibana for end-to-end real-time analytics pipelines.
Professionals aiming to stay ahead in the dynamic data landscape by acquiring comprehensive skills in real-time data processing and visualization.

Apache Kafka v3 for Big Data Streaming - Python Hands On

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 2min

Introduction to Data Ingestion3 lectures • 9min

Introduction to Apache Kafka2 lectures • 4min

Kafka fundamentals4 lectures • 10min

Kafka CLI hands-on8 lectures • 18min

Kafka Hands-on in python6 lectures • 15min

Real World Big Data Ingestion Python Project: Streaming Twitter Data with Kafka10 lectures • 33min

Real Time Streaming pipeline Handson : Kafka, Flink, ElasticSearch and Kibana10 lectures • 59min

Real World Project Exercice-Solution0

Bonus - Optional Lectures3 lectures • 12min

Requirements

Description

Who this course is for: