Modern Event-Driven Spring Boot: Kafka & Spring Cloud Stream

Set up kafka locally with docker compose for learning and testing, using the standard jvm-based kafka image with command line tools, and note the native graalvm image for integration testing.

Explore how linger.ms and batch.size govern when the Kafka driver sends producer messages, including console producer behavior, shaping wait time, batch size, and immediate sends when linger.ms is zero.

Learn how Kafka log retention policies control how long data stays on disk, with default seven days, configurable by time or size in server.properties, and periodic cleanup checks.

Learn how consumer groups in Kafka ensure only one instance of a microservice processes a given order event by joining as a single logical consumer.

Demonstrates listing consumer groups in a Kafka environment by running a command inside the Kafka container, revealing groups such as payment service and inventory service.

Learn about anonymous consumer groups in Kafka, where starting multiple console consumers assigns random group names and delivers messages to each group, while inactive groups are removed after seven days.

Understand how the client library uses the key to determine the partition and submit messages to Kafka. Override by providing the partition manually if needed.

Observe how Kafka performs partition rebalancing when a new consumer joins the payment service group, redistributing two partitions as consumers join and leave.

Explore how partitions and consumer groups scale in Kafka through auto scaling and partition rebalancing. A partition is assigned to one consumer, never shared, preserving message ordering.

Create a topic with two partitions and produce five messages. Start two consumers in a group to track offsets, observe log end offsets, current offsets, and lag as they rebalance.

Generate a Spring Initializr Maven project named Event driven playground with Spring Cloud Stream and Apache Kafka, then set up section 01 with its own runner and YAML config.

Create a Kafka consumer with Spring Cloud Stream by exposing a bean via a consumer config class and logging messages from the demo topic.

Explore reactive consumer design with Spring Cloud Stream and Kafka, using flexofty and monofty publishers, and return mono of void for framework-driven subscription with back pressure.

Create a simple Kafka producer that supplies sequential messages to a Kafka topic while a consumer subscribes, illustrating a two-application demo with producer and consumer packages.

Demonstrates running separate consumer and producer apps as two JVMs, wiring beans via section runners and YAML bindings to a single demo topic, then adjusting polling intervals to control throughput.

Demonstrate the limitations of poller-based suppliers and show how returning a flux publisher enables reactive order events and product view events to Kafka without polling.

Create a reactive Kafka producer by returning a flux from a supplier bean, emit data on demand, and publish messages to Kafka using a sync-based flux or a stream bridge.

Set up three consumer classes across separate JVMs and a single producer, with a Spring component counter incremented per message and printed on shutdown, and millisecond emission via YAML.

Map a flux of orders to flux of payments in a one-to-one reactive processor. Filter physical products for shipments and use flatMap for one-to-many notifications to Kafka, with Spring subscribing.

Explore dynamic routing that decides at runtime whether to route digital deliveries to a digital delivery topic or physical deliveries to FedEx or USPS topics based on carrier availability checks.

Inspect three-broker kafka cluster, create a topic with two partitions and replication factor three, and verify leaders and in-sync replicas while simulating broker failures to ensure controller election with majority.

Optimize producer side batching to boost throughput in Kafka with Spring Cloud Stream by tuning linger.ms, batch.size, and optional compression (LZ4) to batch and send messages efficiently via stream bridge.