Apache Kafka for absolute beginners in Java

Learn Apache Kafka the way data and AI engineers use it: hands-on, in Java, on a real multi-broker cluster.

Why learn Kafka, and how to learn it

Apache Kafka is the backbone of real-time data. It moves events between services, feeds data pipelines, and keeps machine learning models and AI applications supplied with fresh data instead of yesterday's batch. Many engineers use Kafka for years without knowing what happens inside it. They copy a configuration, hope for the best, and struggle when messages are duplicated, delayed or lost.

This course takes the opposite approach. You will run a 3-broker Kafka cluster in Docker on your own computer, write Java producers and consumers, and then break things on purpose. You will stop a broker, trigger a rebalance, send a bad message and replay old events, so you understand why Kafka behaves the way it does. Concepts come first, then a lab proves them.

What this course covers

• Batch vs stream processing, and why Kafka sits at the center of real-time data architecture

• Setting up a 3-broker Kafka cluster with Docker, and working with the Kafka command-line tools

• Topics, partitions, offsets and how Kafka stores and orders data

• KRaft, the controller, replication, leaders, followers and the in-sync replica set (ISR)

• Log segments, retention policies and log compaction

• Failure labs: what happens when a leader broker or a follower broker goes down

• The Java Producer API: keys, serialization, delivery callbacks, poll() and flush()

• Partitioning strategies, including custom partitioners

• Reliable delivery: retries, idempotent producers, acks and min in sync replicas

• Producer tuning with linger ms, batch size and compression

• The Java Consumer API: the poll loop, group id, auto offset reset and heartbeats

• Consumer groups, group coordination and scaling consumers with partitions

• Rebalancing: eager vs cooperative-sticky assignment

• Safe offset management, manual commits and dead-letter topics for poison messages

• Reprocessing data by resetting consumer group offsets, and by building a replay tool that seeks to a point in time

• A capstone project: build a real-time order pipeline with two producers, an order loader with a dead-letter topic, a sales aggregator and a replay tool, then test it against crashes and broker failures

What you will take away

• A clear mental model of how Kafka works internally, not just how to call its API

• The ability to explain partitions, offsets, replication and the ISR in an interview or a design review

• A working 3-broker Kafka lab that you can reuse for your own experiments

• Production-minded Java producers that deliver messages reliably and efficiently

• Java consumers that scale with consumer groups and handle failures and bad messages safely

• The skill to diagnose common problems such as consumer lag, rebalances, duplicates and timeouts

• The confidence to reprocess historical events when a bug or a new requirement demands it

• A strong foundation for building real-time pipelines that feed analytics, machine learning and AI applications

• A working, tested real-time pipeline that you built yourself, which you can show in your portfolio or discuss in interviews

Is this course for you?

You need basic Java and some comfort with a terminal. You do not need any Kafka, Docker or Python experience, because every setup step is explained.

This course focuses on core Kafka: brokers, topics, producers and consumers. Kafka Streams, KSQL DB, Kafka Connect, Schema Registry and security are not covered.

If you want to understand Kafka well enough to build with it, fix it when it misbehaves and explain your design choices, this course is for you. Enroll now and start with a running cluster in your first chapter.