Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Apache Kafka Architecture
Rating: 4.2 out of 5(136 ratings)
7,967 students

Apache Kafka Architecture

Get Ready For Kafka Job Interview: Consumer Groups, Replication, Batching, Compressing, Log Compaction, ISRs and more
Created byAnatolii Stepaniuk
Last updated 5/2021
English

What you'll learn

  • Get Ready for Kafka JOB INTERVIEW!
  • Fully Understand Architecture of Apache Kafka
  • Embrace Consumer Group Abstraction
  • How partitions are assigned to a group of consumers?
  • Understand how data is replicated in cluster
  • What is Delete Cleanup Policy and how to calculate correct config values?
  • How Log Compaction can help reduce disk usage?
  • How to batching of messages together on Producer and Consumer sides?
  • When is a good idea to use Compression in Kafka?
  • What role Controller plays in Kafka Cluster?
  • Deep understanding of Rebalance Protocol
  • How Static Membership can avoid undesired rebalances?
  • Incremental Cooperative Rebalance and its benefits over Eager Rebalance

Course content

5 sections • 25 lectures • 2h 32m total length

  • Topics and Records6:24

    Discussing overall Kafka Architecture:
    Topic is the abstraction we write and read from.
    It is split into partitions that can be handled by separate brokers.
    Partition is Log (a folder with Segment files).

    Music: «Creative Minds» from Bensound.com

  • Kafka Files Structure10:29

    We are going to investigate the structure of real Kafka files and compare that to our existing knowledge.
    We are going to run Kafka on our local machine in Docker!

    Music: «Creative Minds» from Bensound.com

  • Processing Order6:41

    Discussing ordering guarantees provided by Kafka.

    Music: «Creative Minds» from Bensound.com

  • Consumer Groups9:03

    Discussing Consumer Group abstraction - a way to scale our Consumer's performance without duplicated processing.

    Music: «Creative Minds» from Bensound.com

  • Replication6:36

    Discussing how Kafka replicates data across the cluster for recovery in case of broker's outage.

    Music: «Creative Minds» from Bensound.com

  • Controller6:22

    Discussing a role of Controller and how it is elected.

    Music: «Creative Minds» from Bensound.com

Requirements

  • General Computer Science Knowledge

Description

Hi there!

You want to prepare to your Kafka Job Interview?
Or just want to know how Kafka works inside?

You are in good company!
My name is Anatolii and I am a Software Developer in the Internet of Things.
Every second we receive tons of data from our sensors and we've chosen Kafka as a backbone of our distributed backend application.

In this course, I am using my practical experience of running Apache Kafka in production and describe in detail the Architecture of Kafka and the motivation behind it.

This course is for you if:

  • you want to prepare for a Kafka Job Interview questions

  • you want to make better architectural decisions for your messaging system

  • more easily debug production issues with Kafka

  • or just want to know how Kafka is built inside.

    We will cover:

  • Consumer Groups and rebalancing of partitions across Consumers

  • Replication of data in Kafka for redundancy and recovery

  • Different options to clean space in Kafka (delete or compact the logs)

  • Batching and compressing messages

  • Fault tolerance in Kafka Cluster

  • Rebalance protocol

  • Static membership

  • Incremental Cooperative Rebalance

We will also run Kafka in Docker to investigate Kafka files and their structure.

In this dense course, I've combined information from these primary sources:

  • The actual Kafka code

  • Kafka Improvement Proposals (KIP) - internal discussions in the Kafka community that describe the feature before its implementation

  • and of course, the knowledge I received personally from running Kafka in High Load production

If you want to know how Kafka is built or get prepared for your Kafka Job Interview I am glad to see you on the course!

Who this course is for:

  • You want to prepare to Kafka Job Interview Questions
  • Interested in how Kafka works
  • You need to evaluate if Kafka is a good fit for your project
  • You want to learn design of modern Scalable Distributed Systems

Report abuse