Apache Kafka for Distributed Systems

Install and run Apache Kafka with Zookeeper on Windows, macOS, and Linux using Docker and docker compose, configuring images, containers, and environment variables.

Connect to Apache Kafka using two GUI tools, Kafka UI and Offset Explorer, via docker compose. Install, configure, and test connections to Zookeeper and brokers without the command line interface.

Install and configure Apache Kafka across Windows, WSL2, and Docker, noting native Windows limitations, Java JDK requirements, topic and segment deletion issues, and access via Kafka UI and Offset Explorer.

Explore the core concepts of Apache Kafka, including producers, brokers, clusters, and consumers, and see how asynchronous microservice communication uses messages to enable loosely coupled data flows.

Cover cluster and brokers in Apache Kafka by showing how multiple brokers enable parallel requests, vertical and horizontal scaling, and high throughput, with each broker storing part of data.

Explain why multiple brokers in a Kafka cluster improve scalability, fault tolerance, and load balancing through data replication and durability, with hands-on Docker Compose deployment and Kafka UI.

Simulate a real-world scenario with Kafka topics and partitions; account and customer microservices publish account and customer events to topics, while the report microservice consumes and a third-party service subscribes.

Understand how Kafka broker discovery uses bootstrap servers and cluster metadata to route producers and consumers to the right brokers, with topics and partitions, across the cluster.

Learn to manage kafka topics via the topics cli: create, list, and describe topics with a bootstrap server, and understand partitions and replication.

Learn how Kafka partitions define topic parallelism by configuring num.partitions in server.properties. Topics are created with the defined partition count and can be validated by describing the topic.

This lecture explains how Kafka consumers fetch from nearest partitions or replicas starting in version 2.4, reducing latency, improving scalability and availability through nearest replica access.

Demonstrates configuring topic replication factor with Kafka CLI on a multi-broker Docker setup. Shows creating topics with replication factor two and handling errors when replication factor exceeds available brokers.

Review key kafka concepts like clusters, brokers, replication factors, and leader partitions; explain metadata exchange and how producers and consumers interact with leaders and nearest replicas.

Learn how Kafka producers trigger data creation, produce messages with optional keys to specific partitions using round robin or hashing, and leverage broker metadata for efficient batching.

Master the Kafka console producer on the CLI to create topics, produce messages (with and without keys), and manage case sensitivity, partitions, and replication factors across brokers.

Learn how the producer serializes data into byte arrays before sending to Kafka, using string, JSON, Avro, or Protobuf serializers; understand broker storage and consumer deserialization.

Explore producer partitioner strategies in Apache Kafka, including default, round robin, sticky, and custom partitioners, and how keys, hashing, batch size, and linger settings influence partition distribution.

Learn practical kafka consumption via the command line: create topics, produce to partitions, and read from beginning or specific partitions and offsets to observe ordered versus unordered delivery.

Explore how consumer delivery semantics ensure reliable data processing in Kafka, detailing at most once, at least once, and exact once within partitions, consumer offsets, and batches.

Kafka provides partition distribution strategies, including round-robin, default partitioner, and uniform sticky partitioner, plus rack awareness and leader balancing, distributing partitions across brokers based on partition count and keys.

Explore building a .NET console app and an ASP.NET Core microservice to demonstrate producer and consumer patterns with Apache Kafka, using Confluent.Kafka, and event-driven messaging with key/value typing.

Explore producer retries in Kafka, including how acks (1 and -1) affect broker acknowledgement and how retry backoff milliseconds, message send max retries, and request timeout milliseconds are configured.

Master primitive serialization in Apache Kafka by using the ISerializer for keys and values, and switch to IAsyncSerializer for complex types such as protobuf, avro, or json.

Configure Kafka consumers in distributed systems by mastering consumer configs, client ID, and group IDs with static membership. Explore SSL and SASL security options.

Create a custom async deserializer for Apache Kafka by implementing IAsyncDeserializer to deserialize json data asynchronously using System.Text.Json, enabling producer and consumer data processing.

Configure advanced consumer settings by managing the heartbeat thread and pull thread, setting the heartbeat interval, session timeout, max poll interval, and fetch min and max bytes for group coordination.