Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Mastering Apache Kafka: From Beginners to Advanced
Rating: 4.1 out of 5(8 ratings)
112 students

Mastering Apache Kafka: From Beginners to Advanced

Learn Apache Kafka comprehensively from fundamentals to advanced topics and practical projects in this master course.
Created byEDUCBA Bridging the Gap
Last updated 11/2024
English

What you'll learn

  • Fundamentals of Apache Kafka and its architecture.
  • Installation and setup of Kafka and Zookeeper.
  • Creating Kafka topics, producers, and consumers.
  • Customizing producers and consumers for specific use cases.
  • Handling Kafka broker scenarios and cluster management.
  • Configuring Kafka properties for performance optimization.
  • Understanding Kafka internals like buffer memory and batch size.
  • Managing Kafka consumer groups and load balancing.
  • Implementing real-world projects using Kafka and Spark for data streaming and aggregation.
  • Troubleshooting common issues and ensuring high availability in Kafka clusters.

Course content

3 sections • 62 lectures • 9h 27m total length

  • Introduction to Apache Kafka9:13
  • Zookeeper and Kafka Installation13:25
  • Zak and Kafka Installation Steps for Windows6:48
  • Creation of Topics13:03
  • Console Producer and Producer Consumer9:41
  • Partitions and Replication Factor9:02
  • Implications of Kafka and ZK11:51
  • Implications of Kafka and ZK Continue4:35
  • Rebalancing of Kafka Nodes6:20

    Start a producer and consumer on topic test one, publish messages, and observe how Kafka rebalances partition leaders across brokers to maintain high availability when a broker goes down.

  • Bring Down All Zak Nodes7:55
  • All Kafka Nodes6:20
  • Kafka Custom Producer Part 111:57
  • Kafka Custom Producer Part 26:48
  • Kafka Custom Producer Part 38:14
  • Kafka Custom Consumer Part 111:23
  • Kafka Custom Consumer Part 27:38
  • Kafka Custom Consumer Part 313:00
  • Kafka-Broker-Scenarios Part 19:15

    Explore Kafka broker and topic behavior by corrupting the leader broker's topic file and observing producer and consumer results.

  • Kafka-Broker-Scenarios Part 29:27
  • Kafka-Broker-Scenarios Part 312:44
  • Kafka-Broker-Scenarios Part 49:38
  • Kafka-Producer-Scenarios Part 19:01

    Explore how multiple producers send to the same topic and verify that Kafka preserves message sequencing at the consumer, with producers posting messages in random order but consumed in order.

  • Kafka-Producer-Scenarios Part 29:01
  • Kafka-Producer-Scenarios Part 314:53
  • Buffer .Memory Property Manipulation8:25
  • Buffer .Memory Property Manipulation Continue9:23
  • Batch .Size Property Manipulation8:31
  • Batch .Size Property Manipulation Continue8:28
  • linger .MS Property Manipulation10:22
  • Group.id Property Manipulation9:58
  • Consumer Groups10:27
  • Impact of Broker not Available on Prod8:15
  • Impact of Broker not Available on Prod Continue5:52
  • Kafka Brokers11:47
  • Kafka Brokers Continue11:44
  • Impact of Zak Instance7:49
  • Impact of Zak Instance Continue7:04
  • Load Balancing Part 18:36
  • Load Balancing Part 27:02
  • Load Balancing Part 310:06
  • Load Balancing Part 49:36
  • Load Balancing Part 510:35

Requirements

  • Basic understanding of Linux command line interface.
  • Familiarity with Java programming language.
  • Knowledge of distributed systems concepts (beneficial but not mandatory).
  • Understanding of fundamental concepts of messaging systems (e.g., queues, topics).
  • Basic knowledge of Hadoop ecosystem components (like HDFS) would be advantageous.

Description

Course Introduction: Welcome to the comprehensive course on Apache Kafka, designed to take you from a beginner to an advanced level in utilizing Kafka for scalable and efficient data streaming solutions. Whether you're new to distributed systems or looking to enhance your knowledge, this course covers everything from the basics of Kafka and Zookeeper setup to advanced topics like producer and consumer configurations, fault tolerance, and real-world project implementations. By the end of this course, you will have the skills and confidence to leverage Apache Kafka effectively in various applications, ensuring robust data handling and real-time analytics.

Section 1: Apache Kafka - Beginners to Beyond

This section provides a comprehensive introduction to Apache Kafka, covering the foundational aspects of setting up Kafka and Zookeeper, essential for managing distributed systems. Students will learn about topic creation, using both console-based producers and consumers, and delve into advanced concepts such as partitions, replication factors, and Kafka node rebalancing. The lectures also explore custom producer and consumer configurations, addressing various scenarios and performance optimizations through buffer memory, batch size, and linger time manipulations. Participants will gain insights into managing consumer groups, handling broker failures, and implementing effective load balancing strategies.

Section 2: Project on Apache Kafka - Top Trending Songs

In this project-oriented section, students will apply their Kafka knowledge to a real-world scenario of analyzing top trending songs. Starting with the installation of Kafka and Spark, participants will set up the project structure and develop Scala objects for data aggregation. The course progresses with practical demonstrations on data production using Kafka producers, consuming data, and writing data to Kafka topics. By the end, learners will be equipped with the skills to implement and optimize data streaming solutions using Apache Kafka in a project context.

Throughout this journey, you gain a deep understanding of Kafka's architecture, its role in data streaming, and practical skills in setting up, configuring, and optimizing Kafka environments. From learning essential concepts like topics, partitions, and replication to mastering advanced techniques such as managing producer and consumer performance, you will be equipped to tackle complex data streaming challenges. Keep exploring and applying Kafka's capabilities to innovate and streamline data workflows in your projects.

Who this course is for:

  • Software Developers: Those interested in learning about Apache Kafka for building real-time data pipelines and streaming applications.
  • Data Engineers: Professionals aiming to integrate Kafka into their data architecture for scalable and efficient data processing.
  • System Administrators: Individuals responsible for managing and deploying Kafka clusters in enterprise environments.
  • Big Data Professionals: Those working with Hadoop ecosystem tools and seeking to incorporate Kafka for data ingestion and processing.
  • Tech Enthusiasts: Anyone curious about distributed messaging systems and real-time data processing technologies.

Report abuse