Master Apache Kafka With Spring boot Java and Testing

Explore how Kafka topics represent the fundamental unit of events, using append-only logs for durable, immutable data and enabling topic-based organization and multiple consumers.

Partitioning splits a topic into partitions across nodes; without a key, messages round-robin; with a key, similar keys land in same partition, offsets ensure order within partitions but not across.

Learn how Kafka brokers in a distributed cluster host topics and partitions, support client read/write, and use replication for fault tolerance.

Understand how partition replication provides fault tolerance by maintaining leaders and in-sync followers across brokers, with automatic leader election and data flow from leader to ISR.

Learn how kafka producers publish messages to topics and partitions, route by key using partitioning, connect to the leader, and configure acknowledgements for durability.

Explore how Kafka consumers read data from topics and partitions, how consumer groups share partitions and preserve ordering, and how offsets enable semantics from at most once to exactly once.

Explore how distributed systems use brokers, replication, and partitioning to store and propagate data, while covering service discovery, consensus, and Zookeeper in Kafka.

Zookeeper coordinates the kafka cluster by managing ephemeral broker nodes, handling register and heartbeat events, and electing a controller to assign partitions and leaders.

Download and install Docker from the official site, choosing Docker Desktop for Windows or macOS. Learn how to start Kafka clusters and ZooKeeper using Docker in upcoming lectures.

Start a single-node Kafka cluster with docker-compose by defining Zookeeper and Kafka broker services using Confluent images, configuring client ports, broker ID, and advertised listeners.

Add three environment variables to the Kafka broker for replication: topic replication factor, transaction state log replication factor, and transaction state log min.isr to 1.

Set up a Spring Boot project with Gradle, Java 17, and add Spring Web and Kafka dependencies; generate, build, and prepare a Docker Compose for a Kafka cluster.

Connect Spring Boot apps to a Kafka cluster, configure bootstrap servers, and create a topic via code using a topic builder with three partitions and a single replication factor.

Learn to build a simple Spring Boot Kafka consumer, configure key and value deserializers, and subscribe to a topic to log received payloads.

build a Kafka producer by configuring serializers and a Kafka template, then publish messages to a topic using an application runner that generates fake data.

design generic events that wrap diverse payloads in a flexible event structure, leverages a generic event type and object mapper for serialization, and uses customer IDs as keys for ordering.

Discover how to read the key from Kafka headers on the consumer side and read event meta data from producer records, including custom headers like trace ID for end-to-end tracing.

Deserialize a generic customer event payload in a Kafka consumer with an object mapper and a type reference, and implement error handling that throws on failure to trigger retries.

Explore how Spring Kafka consumer groups guarantee in-order processing by routing messages with the same key to the same partition, using three fixed customer IDs to illustrate key-to-partition mapping.

Learn how to produce events with a key that routes to a fixed partition, ensuring the same customer ID always lands in the same partition.

Scale consumer throughput by running multiple instances in a group with concurrency, observe partition assignment and rebalancing, and ensure the same key maps to the same partition for ordering.

This lecture explains the default error handling and retry behavior in Spring for Kafka, including error topics, multiple retries, and offset commits after failures.

Explore implementing a Kafka listener error handler in Spring Boot, including a custom error handler bean, logging, and returning messages to a failed topic for recovery.

Explore configuring the global default error handler in Spring Kafka, including fixed and exponential backoffs, max retries, and how after-handle behavior decides offset commits and event reprocessing.

Create a concurrent Kafka container factory wired to the consumer factory and enable the delivery attempt header. Leverage error handlers and use delivery counts to guide offset commits and recovery.

Configure Kafka's default error handler to distinguish retriable and non-retriable exceptions in a Spring Boot application, avoiding unnecessary retries for non recoverable errors by classifying readable versus not readable exceptions.

Set the auto-offset-reset property to determine where a Kafka consumer starts when no committed offset exists. Choose earliest or latest, and ensure offsets are committed to avoid repeated restarts.

Learn how to pause partitions assigned to a consumer after delivery counts reach the max retry threshold, using force, posting assignments, and resuming on rebalance to handle bad events.

Learn to manually commit Kafka offsets in spring boot using manual immediate acknowledgement, configure the listener, and commit current and prior offsets when processing succeeds.

Implement blocking retries in a Kafka consumer by processing one event at a time, using manual immediate acknowledgement, and committing the offset only after successful processing, with backoff.

Master non-blocking retries and death letter topics in Kafka with Spring Boot, retry topics and interval headers. Understand when to apply blocking versus non-blocking retries and how ordering affects correctness.

Implement non-blocking retries in Kafka with Spring Boot by configuring topic-based backoff, retry topics, and delta topics, while monitoring via headers, logs, and group IDs.

Learn to implement a dead letter topic (dlt) handler in a Spring Boot Kafka app by annotating a method in the same class, logging payloads, and aligning instances with config.

Set up a new Spring Boot project for transactions, configure Kafka producers and consumers, and implement atomic operations on both Kafka and the database using an in-memory database and JPA.

Copy the Kafka setup to a new project and mirror the app config in the RC environment. Ensure transaction handling is aligned for seamless integration.

Explore why transactions matter in Kafka, preventing duplicates from producer retries and achieving exactly-once pipeline processing, including zombie instances and fencing concepts.

Explore atomic transaction flows in Kafka, enabling begin and commit across topics and offsets, while using a transaction coordinator and commit markers to prevent zombie fencing in Spring Kafka.

Configure transactional producers in Spring Boot for Apache Kafka with a transactional id, enabling atomic writes across two topics. Use a transaction manager and adjust consumers to read committed data.

Explore exactly once processing in Spring Boot Kafka using transactional producers and a consumer that reads only committed messages, writing to multiple topics within one transaction.

Coordinate database transactions with Kafka transactions for atomic processing using a transactional producer and JP transaction manager, and implement idempotent handling via an inbox table of event IDs.