Mastering Apache Kafka: From Beginners to Advanced

Explore how Apache Kafka enables end-to-end messaging and streaming with brokers, producers, consumers, topics, and Zookeeper. See its open-source nature, ELK stack compatibility for visualization, and clustering for data availability.

Install kafka and its components by setting up zookeeper as the heart of kafka and creating a three-node cluster. Configure data and logs, zoo.cfg, myid, and start zookeeper with zkServer.cmd.

Master the Windows installation of zookeeper for Kafka by configuring Java_HOME and launching a three-node zookeeper cluster. Troubleshoot port 2181 and config issues to bring all nodes online.

Set up a three-node Kafka cluster with unique broker IDs and separate log directories, then create a topic with the Kafka topics batch file using bootstrap servers 9093, 9094, 9095.

Learn how topics are partitioned into partitions, apply replication factor for high availability, and manage topics with list and describe commands.

Learn how Kafka clusters designate a leader per partition to manage producer messages and replicate to in-sync replicas (ISR) for high availability.

Explore how Kafka and ZooKeeper handle failures, from node outages to broker downtime, through leader rebalancing, partitions, and replication for high availability.

Explore how kafka topic creation, partition counts, and replication factors work, including why replication cannot be changed post-creation and how to alter partitions with json-based configurations.

Start a producer and consumer on topic test one, publish messages, and observe how Kafka rebalances partition leaders across brokers to maintain high availability when a broker goes down.

Demonstrates how Kafka servers behave when all Zookeeper nodes go down, with ISR expanding and producers and consumers continuing to operate, even as topic creation tests report errors.

Explore how Kafka nodes rely on zookeeper to register topic metadata, handle timeouts, and maintain cluster health; learn how zookeeper failures affect topic creation, broker availability, and recovery.

Kafka custom producer part 1 teaches how to build custom producers using the Java API for Kafka, detailing required properties like bootstrap servers, key and value serializers, acks, and retries.

Develop and run a custom kafka producer that continuously sends string key/value records to a topic in an infinite loop, using producer and producer record with runtime confirmations.

Compile and run a custom Kafka producer against a demo topic, with Zookeeper and broker setup. Test end-to-end data flow by consuming produced messages with a Kafka console consumer.

build a java kafka custom consumer by configuring bootstrap servers and string deserializers, enabling auto commit and session timeout, then subscribe to a topic with a mandatory topic name.

Explore how a Kafka consumer reads topic data in an infinite loop using poll, holds records, formats and prints offsets, keys, and values, and imports the consumer package.

Explore building and testing a kafka custom consumer in java, including topic subscription, group id configuration, and end-to-end producer–consumer interaction.

Explore Kafka broker and topic behavior by corrupting the leader broker's topic file and observing producer and consumer results.

Delete the leader topic file to observe behavior as producers publish and consumers consume, then explore corruption of a non-leader broker and its impact on failover and replication.

Demonstrate Kafka's durability and leader rebalancing by running a producer and consumer, simulating broker failures and topic file corruption or deletion across non-leader and leader brokers.

Explore how Kafka brokers manage leader changes and in-sync replicas during failover, preserving data integrity. See broker termination during active read-write to assess producer and consumer resilience.

Explore how multiple producers send to the same topic and verify that Kafka preserves message sequencing at the consumer, with producers posting messages in random order but consumed in order.

Kafka preserves sequencing and offsets across producer restarts, letting the same topic yield new messages from the correct offset. A custom consumer remains open, preventing re-reading and ensuring no duplication.

Explore how a Kafka producer behaves when the cluster is down, with broker failures and leader loss, and learn that at least one broker must stay up for successful publishing.

Tune producer buffer memory and other configurations in Apache Kafka. Manage ZooKeeper ensembles and Kafka brokers, create topics with partitions and replication factors.

Examine how to verify topic creation and partitioning, then publish messages with a producer while tweaking producer properties such as bootstrap.servers, compression, buffer.memory, batch.size, and linger.ms.

The lecture demonstrates using producer.config to override defaults, showing how buffer.memory between the producer and topic restricts message size and how raising it from 3000 to 4500 fixes serialization errors.

Experiment with batch.size and buffer.memory to see how a 4500 byte limit triggers java.lang.IllegalArgumentException when allocating 10000 bytes, then raise buffer.memory to 45000 to enable larger batches.

Explore how kafka consumer properties and group.id establish consumer groups, and how offsets distribute messages among group members rather than to every consumer.

shows how Kafka topic is divided into partitions under a consumer group and how partition-to-consumer mapping determines which consumer reads each message.

Explore how linger.ms, batch.size, and buffer.memory shape producer batching and when messages posted to topics. Observe consumers reading from the beginning and the impact of gzip and snappy on delivery.

See how broker failures affect Kafka producers and consumers. Topic test has leader broker two with replicas two and three; stopping broker two halts the producer while the consumer continues.

Explore how broker outages affect producers and consumers in Kafka when leaders go down and acks matter. Consider migration from other tools and testing negative scenarios.

Learn how shutting down Zookeeper peers affects Kafka brokers and cluster communication, triggering connection reset errors and logs, and test Kafka's resilience by creating a topic after outages.

Understand kafka brokers and bootstrap servers on localhost: 9093, 9094, and 9095 for topic creation. See how zookeeper quorum, an odd-numbered cluster, governs topic availability.

Analyze the impact of killing zookeeper instances on the Kafka producer and consumer, and observe message delivery on topic test as zookeeper instances are terminated.

Explore how increasing producer load affects producer and consumer synchronization in Apache Kafka, including Zookeeper failures, topic description, replicas, and leader elections after restart.

Simulate heavy load on Kafka by producing a ~20 kb message one million times and consuming it to observe broker and ZooKeeper behavior under failures.

Restart Kafka servers, compile and run the producer code, and create a topic with one partition and replication factor three to validate message delivery and load balancing.

Start a simple producer and consumer, compile and run demos, create topics, and examine how messages, keys, values, and offsets are consumed under load.

Explore load balancing part 4 in Kafka by running producer and consumer across topics, observing offsets, keys, and values, and testing broker failures to reveal data loss risks and leader failover.

Explore Kafka load balancing and fault tolerance as producers reach high throughput while brokers and leaders failover, showing how replicas maintain data integrity and consumer resilience.