The Complete Apache Kafka Practical Guide

Install Apache Kafka on Mac by downloading the binary (version 2.4.0) from mirrors, extracting it into a Kafka folder, and ensuring Java is installed.

Discover how to install Apache Kafka on Windows for practice. Understand script extensions and the difference between Windows and Unix-like systems, then use an Ubuntu virtual machine.

explain how to inspect the Apache Kafka installation folder, identify script and configuration files, and start Kafka services, noting that a broker needs Zookeeper and reviewing startup logs.

Learn to observe the Kafka folder contents, unzip the archive, and run services via bin scripts on Windows or Unix, while inspecting config and logs to understand default parameters.

With Zookeeper up at port 2181, start the Kafka server using server.properties, then confirm broker id 0 and listener on port 9092, while noting log storage in logs and tmp/kafka-logs.

Observe the current Kafka server and Zookeeper setup, inspect the logs directory and tmp/kafka-logs, and learn default broker id zero and port 9092 as you prepare to create topics.

Create the first Kafka topic using the kafka-topics script with bootstrap server and name cities. Observe the cluster logs showing partition city zero and broker zero, confirming default configuration.

Discover how to list and describe a Kafka topic, using zookeeper or bootstrap server, and review partitions, replication factor, and leader and in-sync replicas.

Begin by using your local Apache Kafka cluster to run the console producer and console consumer, then launch multiple producers and consumers in parallel to observe message flow.

Connect to localhost:1992 as the bootstrap server and consume messages from the cities topic using the Kafka console consumer, verifying real-time message flow from the producer.

Use the console consumer from beginning to read all messages from a topic. The Apache Kafka cluster stores messages after consumption, enabling multiple consumers to read them later.

Understand Apache Kafka as a distributed publish-subscribe messaging system, where brokers store messages and serve multiple producers and consumers, with messages stored on disk and support for asynchronous, multi-broker clusters.

Explore how Kafka brokers store messages for producers and consumers, form scalable clusters, distribute messages across brokers, and provide failover in case of broker failure.

Understand how ZooKeeper ensembles ensure kafka cluster availability by using an odd-numbered server set and a quorum of half plus one, preventing split-brain scenarios.

Explore multiple Kafka clusters across regions, each with its own brokers and Zookeeper, and how mirroring enables data synchronization so producers and consumers connect to the nearest cluster, reducing load.

Explore Kafka topics with unique names and immutable offsets, where messages are appended to the log end and cannot be inserted, and older records delete after seven days.

Explore how topics are stored on Kafka brokers and why partitions spread a topic across multiple brokers to boost fault tolerance and scalable writes and reads.

Spread messages across partitions to boost performance by parallel reads and writes on multiple brokers. Replicate partitions to ensure fault tolerance and maintain unique offsets within each partition.

Learn how topics divide into partitions across brokers, where a leader handles writes and followers replicate messages to provide fault-tolerant, replicated storage through a configurable replication factor.

This section introduces the course's GitHub repository, containing all examples and projects you can clone and use. Follow along to create each project yourself and access the diagrams demonstrated earlier.

Explore the diagrams in the Diagrams.html file from the Apache Kafka Course repo, showing how to start broker and zookeeper, manage topics, and copy commands.

Create a multi-partition topic in Kafka, produce messages, and observe unique offset per partition; read from topic or partition and learn why the consumer offsets topic has 50 partitions.

Start Zookeeper and the Kafka broker, create a topic with three partitions named animals, and use the console producer and consumer to observe logs in /tmp/kafka-logs.

See how a three-partition topic distributes messages like cat, dog, and monkey across animals zero, one, and two. Observe that a console consumer may read different orders from each partition.

Read messages from a specific partition using the console consumer by specifying the partition number. Observe how starting offsets zero and one retrieve dog and cat from their respective partitions.

List topics and read details for the animals topic in a Kafka cluster, noting 3 partitions, 1 replication factor, and 1 GB segment size, plus the __consumer_offsets topic.

Start the ZooKeeper and Kafka broker, create a topic named Animals with three partitions and replication factor one, produce messages, and observe spread across partitions with asynchronous consumption.