Apache Kafka for Event-Driven Spring Boot Microservices

Explore how a Kafka broker runs as a Kafka server on computer or cloud, forms a leader and followers cluster, and stores events in topic partitions for producers and consumers.

Explore how each partition of a topic has a leader to handle reads and writes, while followers replicate data for redundancy and fault tolerance, enabling balanced leadership across brokers.

Download Apache Kafka from the official website, choose the latest stable release or an older version if needed, then unzip and move it to your user directory.

Start a single Apache Kafka broker using craft mode to replace zookeeper, generate a cluster ID, format log directories, and launch the server with server.properties.

Prepare configurations and storage for three Kafka brokers, start each with kafka-server-start.sh on Mac, using config/craft/server-1.properties, server-2.properties, and server-3.properties, verify all up with no errors as they come online.

Stop producers and consumers before shutting down Kafka servers to avoid message loss, then run the Kafka server stop CLI script for a graceful shutdown that closes log segments.

Learn to create a new Kafka topic by starting the Kafka server, using the topics script, setting partitions and replication factor, and specifying bootstrap servers.

List topics in your kafka cluster using the topics script with bootstrap server to view all topics. Describe topics to see name, id, partitions, leader, replicas, isr, and segment size.

Learn to produce messages to a Kafka topic with the console producer, connect via bootstrap servers, and understand topic auto-creation versus preconfigured partitions and replicas.

Learn to send Kafka messages as key value pairs using the console producer, enabling key value support and a separator to preserve order for messages with the same key.

Learn to use the Apache Kafka console consumer CLI to read messages from a Kafka topic in your terminal, choosing between new messages or all messages from the beginning.

Learn to run a Kafka consumer that reads only new messages by omitting the from beginning parameter. Observe a producer sending a new message to all subscribers.

Learn how to consume key value pair messages from a Kafka topic by configuring the producer to send key and value, and using consumer options to print key and value.

Explore how a Kafka producer uses synchronous communication style to send an order created event to a Kafka broker via a Spring Boot orders microservice, awaiting acknowledgment.

Explore how a Kafka producer uses asynchronous communication to publish a user logged in event to a Kafka broker after authentication for analytics, keeping the login flow responsive.

Run the microservice, ensure Kafka servers are up, and verify the new topic with Kafka topics describe to confirm three partitions, three replicas, and in-sync replicas.

Create a product service interface and implementation to persist product details and publish a product created event to a Kafka topic, with constructor-based dependency injection in the rest controller.

Create a product created event class in the service package containing the product details and created product id, with a no-args constructor and getters and setters for deserialization.

Publish the product created event to a Kafka topic using Spring's Kafka Template after generating a unique product id with UUID and persisting product details, with options for asynchronous sending.

Explore asynchronous sending of a product created event to a Kafka topic, verify with a console consumer, and observe log messages showing the send flow.

Publish events to a Kafka topic synchronously by awaiting broker acknowledgement to ensure persistence, while comparing approaches using CompletableFuture join and get to toggle between async and sync.

Update the create product rest controller to handle exceptions by returning a custom json error object with a timestamp and message, using a logger and Spring's automatic json conversion.

Log and inspect Kafka producer metadata to monitor and debug message delivery; print topic, partition, offset, and timestamp after a synchronous send.

Demonstrates synchronous publishing of a product created event to Kafka, verifies the producer and a consumer script, and confirms topic, partition, and offset metadata.

Configure the Spring Kafka producer to wait for acknowledgements from all in-sync replicas using spring.kafka.producer.acks to strengthen durability and prevent message loss.

Learn to configure the min.insync.replicas property for Kafka topics, either at creation or by updating an existing topic with Kafka Configs, ensuring two in-sync replicas acknowledge writes.

Explore Kafka producer retries by configuring producer retries to ten and setting retry backoff to 1000 milliseconds, demonstrating repeated attempts until success.

Explore how Kafka producer retries and retry backoff work under broker failures, with two in-sync replicas, three brokers, and ten retries at a one second time interval.

Explore delivery timeout, request timeout, and linger, to optimize producer behavior; delivery timeout governs total send duration across replicas, including retries, while request timeout caps a single broker response.

Study Kafka producer delivery and request timeout by running a three-broker cluster with minimum in-sync replicas to two, then trigger retries until the two-minute timeout and observe a timeout exception.

Configure a Kafka producer in Java code by building a properties map, using a producer factory and a Kafka template, with precedence over application.properties for bootstrap servers.

Enable idempotence for the Kafka producer using the application.properties file, setting enable.idempotence, acks to all, and configuring retries and max in-flight requests per connection to five.

Enable kafka producer idempotence in a spring bean by configuring producer properties in code and loading from application properties, with acks all and max in flight capped at five.

Learn how to build a Spring Boot Kafka consumer that reads from a product created event topic with multiple partitions. Leverage consumer groups for scalable parallel processing.

Create a new Spring Boot project for an email notification microservice using Spring Tool Suite, Spring Starter Project, and Apache Kafka dependencies to act as a Kafka consumer.

Configure the Kafka consumer using the application properties file, specifying bootstrap servers, key and value deserializers, and a consumer group id for product created events.

Create a product created event handler class as a Spring component with a handle method annotated as a Kafka listener to process product created events from topic and log arrival.

Create a new shared core module for Spring Boot microservices, move the product created event into it, and add it as a Maven dependency for both microservices.

Add the core project as a maven dependency to the products and email notification microservices, install locally, update projects, and resolve imports including the product created event.

Configure the Kafka consumer in Java code via a Spring bean and a Kafka consumer configuration class, reading bootstrap servers, deserializers, and group id from application properties.

Create a Kafka listener container factory using the bean annotation and the configured Kafka consumer factory, enabling listeners to receive messages and invoke their methods within a Spring Boot application.

Test the Kafka consumer bean configuration by launching microservices, sending a product creation request, and verifying messages flow to the email notification service.

Configure your Kafka consumer to recover from deserialization errors when reading product created events, preventing endless retries of faulty messages and ensuring the email notification service continues with subsequent messages.

Learn how a deserialization error halts a Kafka consumer when invalid json is published, and preview recovering with an error handling deserializer class in a Spring Boot microservice.

Configure the Kafka consumer to gracefully handle deserialization errors using the error handling deserializer wrapper around the JSON deserializer, allowing continued processing of subsequent messages.

Demonstrate how the error handling deserializer helps a kafka consumer recover from deserialization errors in a spring boot microservice setup. See one-time errors recover and new messages process correctly.