Apache Kafka Series - Learn Apache Kafka for Beginners v3

Explore how Kafka topics act as streams in a cluster. Learn how partitions and offsets control data ordering and immutability, with producers and consumers driving the data flow.

Producers write to topic partitions using keys to route and order messages, with null keys routing round-robin, and a partitioner hashing keys murmur2 to assign partitions.

Learn how Kafka consumers request data from brokers in a pool model, read within partitions by offset, and use key and value deserializers to convert bytes to objects.

Learn how consumer groups distribute reads across partitions and how committed offsets let you resume from where you left off. Explore at least once, at most once, and exactly once.

Explore how Kafka brokers form a scalable cluster, how topics and partitions distribute data across brokers, and how bootstrap servers enable clients to connect to the entire cluster.

See how Kafka replication factor duplicates partitions across brokers, with producers writing to leaders and consumers reading from leaders, and learn about ISR and the Kafka Consumer Replica Fetching feature.

Explore producer acknowledgements in Kafka, comparing acks=0, acks=1, and acks=all, and learn how replication factor affects topic durability and no data loss guarantees.

Explore how Zookeeper manages Kafka brokers, metadata, and leader elections. Learn the transition to Kafka Raft, and why Zookeeper remains essential in production until migration completes.

Explore how kafka krev removes zookeeper with kip-500 to scale to millions of partitions, simplify security, and improve shutdown and recovery times, with production-ready status since 3.3.1.

Summarizes Kafka theory: multi-broker clusters with topics, partitions, replication, leaders, and in-sync replicas, plus producers and consumers, acks, offset topics, and the Zookeeper-to-Kafka shift.

Learn to start Apache Kafka using Docker or binaries, launch Kafka with one command, and use Conduktor UI alongside platform-specific setup for Mac, Linux, and Windows.

Install Java JDK 21 (Amazon Corretto) and Kafka binaries on macOS, then extract and add Kafka bin to PATH to run kafka-topics.sh and other CLI commands.

Start kafka in kraft mode on Mac OS X by generating a cluster uuid and starting the server with the kraft configuration; logs are stored in tmp/kraft-combined-logs.

set up java corretto 21, download and extract kafka binaries on linux, and update your path to run kafka cli commands from anywhere.

Start a Kafka broker on Linux in KRaft mode without ZooKeeper; create a cluster UUID, format the log directories with kafka-storage.sh --standalone, then start Kafka using the server.properties file.

install java jdk 21 on wsl2 ubuntu, download and unzip apache kafka binaries, and configure path to access kafka-topics.sh from anywhere.

Start kafka on Windows WSL2 in KRaft mode by creating a cluster UUID, formatting storage, and launching the kafka server, then monitor logs for version 4.0 and keep window open.

Start a Mac OS X Kafka setup by running Zookeeper server start.sh with zookeeper.properties, then Kafka server start.sh with server.properties, keeping both terminals open.

Start Zookeeper and Kafka on a single machine using the Kafka binaries, with two terminals, zookeeper.properties and server.properties, and optionally customize data and logs directories.

Learn to set up Kafka on Windows non-WSL2, install Java 11, download and extract Kafka 3.1.0, and start Zookeeper and Kafka, with Conduktor as a simpler alternative.

Learn to use the Kafka CLI with bundled binaries, set your path, and run topics commands via the bootstrap server, covering OS-specific executables and zookeeper deprecation.

Learn to manage kafka topics with the topics cli: create, list, describe, adjust partitions, and delete topics across secure clusters using Conduktor or local setups.

Learn to use the Kafka console producer to publish data to topics, with and without keys, observe partitioning behavior, and apply producer properties such as acks and key separators.

Learn to use the Kafka console consumer CLI to consume from the beginning or tail of a topic, observe per-partition ordering, and print keys, values, and partitions.

Explore how Kafka console consumers form a group, how partitions are distributed among group members, and how consumer offsets and from-beginning behavior affect rebalancing and lag.

Explore the Kafka consumer groups command line interface to list, describe, and delete consumer groups, and observe current-offset, log-end-offset, and lag across partitions, and understand partition assignment with multiple consumers.

Learn how to reset Kafka consumer group offsets, using dry run then execute, observe lag changes, and compare CLI versus UI options for safer, faster offset management.

Explore the Conduktor UI for Apache Kafka, from cluster management and topic creation to producing, reprocessing messages, and inspecting keys, values, headers, and timestamps, plus topics and consumer groups.

Explore the Kafka SDK list and learn that the course uses the Java official client library, with community-supported SDKs for other languages and pointers on Conduktor.io/kafka/kafka-sdk-list.

Set up a Kafka project in Java with Gradle in IntelliJ, install Java 11, add kafka-clients and slf4j dependencies, create ProducerDemo, and run Hello world to verify.

Learn how to write your first Kafka producer in Java, configure properties, serialize keys and values, send a record to a topic, and verify with a console consumer.

Send non-null keys to a Kafka topic and observe that the same key always lands in the same partition, demonstrated with a Java producer using id_ keys.

Explain how to implement a graceful shutdown for a java kafka consumer using a shutdown hook, wakeup, and finally close to commit offsets and leave the group.

Explore how a Java consumer behaves in a three-partition topic within a consumer group, observe partition rebalances as new consumers join, and see partition assignments and lag in real time.

Master consumer group management by understanding partition rebalance strategies, from eager rebalances that stop processing to incremental cooperative rebalances that minimize disruption and support static membership with group instance IDs.

Demonstrate enabling cooperative sticky partition assignment for a Java consumer and verify incremental rebalances by setting partition assignment strategy to cooperative sticky assignor.

Explore how the Java consumer auto commit behavior works, with poll triggering offset commits when auto commit is enabled and the five-second interval ensures at least once semantics.

Explore advanced Kafka consumer tutorials, including assign and seek to read from a specific offset manually, a rebalanced listener, and threading, in the advanced folder of Kafka programming tutorials.