Modern Event-Driven Spring Boot: Kafka & Spring Cloud Stream

Set up kafka locally with docker compose for learning and testing, using the standard jvm-based kafka image with command line tools, and note the native graalvm image for integration testing.

Set up a local Kafka server with Docker Compose, start the Apache Kafka image on port 9092, and access the container's working directory to run command line tools.

Explore how to access Kafka cli tools inside a running Docker container, run commands with dot slash, use the Kafka topics tool, and navigate broker, producer, and consumer properties files.

In this section we explore Kafka fundamentals, topics, producers, and consumers, using command line tools to build a mental model of event driven architecture and real-time data flow.

Learn how a Kafka cluster provides high availability and horizontal scalability through multiple brokers and a single active controller, with roles defined by process.roles.

Explore how a Kafka cluster uses leader and followers to replicate order events across brokers, ensuring high availability and no single point of failure.

Learn how a bootstrap server lets a Kafka client discover cluster metadata from a single contact point, with fallback to multiple machines for high availability.

Explore Kafka topics through hands-on commands to create, list, describe, and delete topics such as order events, payment events, and shipping events, using the bootstrap server and topic options.

Produce messages to a topic using the Kafka console producer, specifying the topic and bootstrap server. Observe hello world and subsequent messages being sent for learning, testing, and debugging.

Learn to use the console consumer to read messages from a topic using a bootstrap server, understand its default new messages behavior, and enable reading from the beginning.

Watch how the console producer batches messages by default for one second and how linger.ms lets you set 0, 50 ms, or 100 ms for immediate delivery.

Explore how linger.ms and batch.size govern when the Kafka driver sends producer messages, including console producer behavior, shaping wait time, batch size, and immediate sends when linger.ms is zero.

Kafka uses a pull model, with the consumer requesting messages from the broker and requiring acknowledgments to confirm processing. The maximum pull records setting limits fetch size to avoid overload.

Learn how Kafka stores messages as bytes and delegates serialization to producers and consumers. Configure serializers, use Jackson to convert Java objects to JSON, then to byte arrays.

Learn how Kafka log retention policies control how long data stays on disk, with default seven days, configurable by time or size in server.properties, and periodic cleanup checks.

Explore how Kafka topics use offsets to deliver messages in the order added, treating topics as append-only, immutable structures with offsets starting at zero.

Deliver a hands-on demo of Kafka offsets using docker-compose, creating a demo topic, and side-by-side producer and consumer. Show offsets with print.offset as messages are produced and consumed.

Demonstrates printing the timestamp of produced messages by enabling the print timestamp option in the consumer, showing offset and unix time from the beginning.

this demo shows one producer and two consumers, where both clients receive all messages; it's correct for some services but leads to redundant processing with multiple payment service instances.

Learn how consumer groups in Kafka ensure only one instance of a microservice processes a given order event by joining as a single logical consumer.

Demonstrate that in a consumer group only one consumer handles each message, avoiding redundant processing, while different groups receive all messages.

Demonstrates listing consumer groups in a Kafka environment by running a command inside the Kafka container, revealing groups such as payment service and inventory service.

Learn about anonymous consumer groups in Kafka, where starting multiple console consumers assigns random group names and delivers messages to each group, while inactive groups are removed after seven days.

Kafka guarantees in-order delivery within a partition, an append-only structure, so deposits and withdrawals are processed sequentially, while partitioning enables parallel consumption without breaking event order.

Learn how partitions and message keys solve ordering and scaling in Kafka. Topics split into partitions; use a key to route related events to the same partition, enabling parallel processing.

Understand how the client library uses the key to determine the partition and submit messages to Kafka. Override by providing the partition manually if needed.

Demonstrates creating a two-partition topic in Kafka, describing partitions to reveal partition leadership and distribution, and clarifying how topics are logical while partitions are physical in standalone and cluster setups.

Demonstrate how a topic with two partitions distributes messages across two consumers in a single consumer group, using keys to ensure load distribution and per-key ordering.

Observe how Kafka performs partition rebalancing when a new consumer joins the payment service group, redistributing two partitions as consumers join and leave.

Explore how partitions and consumer groups scale in Kafka through auto scaling and partition rebalancing. A partition is assigned to one consumer, never shared, preserving message ordering.

With a three-partition topic, increasing partitions may change partition calculation for keys and affect ordering; use the alter option or create a new topic v1/v2 and switch producers and consumers.

Demonstrate how null keys affect message routing on a two-partition topic, producing messages without keys and observing that null keys do not round-robin across consumers.

Explore offset tracking in Kafka, where each partition assigns a unique offset starting at zero and consumer groups resume delivery from the last delivered offset across partitions.

Create a topic with two partitions and produce five messages. Start two consumers in a group to track offsets, observe log end offsets, current offsets, and lag as they rebalance.

Clarify how Kafka tracks consumer offsets and uses the from beginning option only on first startup to deliver or skip messages, after which offsets govern delivery.

Learn how Kafka tracks offsets and lag for each consumer group and partitions, reset offsets to re-deliver messages, using shift by, duration, or from beginning, with dry run or execute.

Explore how Kafka enables high-throughput, low-latency event streaming with topics, partitions, offsets, and keys to ensure ordering, scalability, reliability, and efficient consumption via consumer groups.

Explore Spring Cloud Stream, a Spring module for building message-driven microservices using binders like Kafka, with producer, consumer, and processor patterns mapped to supplier, consumer, and function.

Explain how binder drivers connect an application to a messaging system, and how bindings map producer and consumer beams to Kafka topics via Spring Cloud Stream conventions and application properties.

Examine binding configuration in the application YAML for Spring Cloud Stream, covering producer and consumer bindings, bean definitions, group names, and Kafka binder overrides for producer and consumer properties.

Explore a Spring Boot playground that teaches event-driven concepts with Spring Cloud Stream and Kafka binder through independent sections, YAML configurations, and hands-on producer, consumer, and processor ideas.

Generate a Spring Initializr Maven project named Event driven playground with Spring Cloud Stream and Apache Kafka, then set up section 01 with its own runner and YAML config.

Create a Kafka consumer with Spring Cloud Stream by exposing a bean via a consumer config class and logging messages from the demo topic.

Explore a simple consumer in action within a modern event-driven Spring Boot setup using Kafka and Spring Cloud Stream, covering docker-compose reset, topic creation, and anonymous group behavior.

Configure auto offset reset to earliest to start consuming from the beginning of a topic, even with an anonymous group, using yaml and section runner to apply Kafka settings.

Configure the group name in the YAML to join the demo group, enabling committed offsets and ensuring only new messages are consumed after the initial five.

Demonstrates consuming messages from multiple topics using a single consumer binding in Spring Cloud Stream to process web and mobile orders.

Choose one-to-one binding per topic to enable binding-level overrides, simplify consumer configuration, and reuse service logic across topics, avoiding topic-level overrides and production risks.

Explore reactive consumer design with Spring Cloud Stream and Kafka, using flexofty and monofty publishers, and return mono of void for framework-driven subscription with back pressure.

Explore multi-input functions in Spring Cloud Stream, zipping driver and passenger streams to produce a trip, a capability currently available only with reactive programming.

Explore Spring Cloud Stream, a Spring module for event-driven microservices with producer, consumer, and processor roles, using binders, bindings, and YAML configurations to map topics.

Build a simple Kafka producer with Spring Cloud Stream, using the Java functional interface Supplier to send messages to a topic via a polling configuration.

Create a simple Kafka producer that supplies sequential messages to a Kafka topic while a consumer subscribes, illustrating a two-application demo with producer and consumer packages.

Demonstrates running separate consumer and producer apps as two JVMs, wiring beans via section runners and YAML bindings to a single demo topic, then adjusting polling intervals to control throughput.

Build kafka messages with key, payload, headers, and event time using spring's message builder. Producers attach the message type; consumers may read the event or a type.

Demonstrate producing messages with keys using a Spring Boot producer config, building messages with payloads and Kafka headers, then consume and access the received key and payload.

Configure and run a Kafka producer and consumer with explicit key serialization, using string serializer and deserializer to ensure correct partitioning and payload handling.

This demo shows producing messages with keys using a serializer in a two-partition topic, illustrating payload, key, trace id, timestamp, headers, and how Spring can deliver payload directly.

Demonstrate the limitations of poller-based suppliers and show how returning a flux publisher enables reactive order events and product view events to Kafka without polling.

Learn to produce messages dynamically with Spring Cloud Stream's StreamBridge, wiring binding names for product view and order events, and sending to topics for integration testing.

Demonstrates streaming command output to Kafka using stream bridge, sending ping results to a demo topic via a ping out destination, with a producer runner and a consumer.

Create a reactive Kafka producer by returning a flux from a supplier bean, emit data on demand, and publish messages to Kafka using a sync-based flux or a stream bridge.

Learn to build a Kafka producer with Spring Cloud Stream using a supplier bean, configure destinations, and create messages with payload, optional keys, and metadata.

Demonstrate scaling of consumers with consumer groups in Kafka using Spring Cloud Stream. Observe three partitions distributed among consumers, rebalance on joins or leaves, and producer throughput per second.

Set up three consumer classes across separate JVMs and a single producer, with a Spring component counter incremented per message and printed on shutdown, and millisecond emission via YAML.

Demonstrate scaling and partition rebalancing in a Kafka setup by creating a three-partition topic and observing partition assignment across consumers.

Clarify how Kafka distributes partitions and handles consumer group changes, showing the system's behavior beyond application concerns.

Build a processor in an event-driven Spring Boot app by consuming events and emitting new ones, exploring one-to-one, zero-or-one, one-to-many, and router patterns.

Emit order objects via a producer to the order events topic, map each to a payment event with a one-to-one processor, and print events to the console for validation.

Activate the shipment processor in Spring Cloud Stream to filter order events, emitting shipment events only for physical products and returning null for digital ones.

Demonstrates a one-to-many mapping processor that emits two notification events per order—sms and email—by returning a list of messages and signaling that each item is a separate event.

Map a flux of orders to flux of payments in a one-to-one reactive processor. Filter physical products for shipments and use flatMap for one-to-many notifications to Kafka, with Spring subscribing.

Explore processor patterns in spring cloud stream, including mapping, filter, and one-to-many processors; compare map, filter, and flat map operators in java stream and reactive pipelines; synchronous vs reactive behavior.

Explore how a processor routes order events to digital or physical delivery using content-based routing rules, creating delivery objects, and sending messages to the appropriate topics.

Explore how to route a single input to two topics in Spring Cloud Stream using stream bridge or message header, with custom bindings and header-based delivery.

Demonstrate content-based routing by sending order objects to digital or physical delivery topics; build digital delivery with order id and email, or physical delivery with order id and address.

Demonstrate content-based routing with Kafka and Spring Cloud Stream by routing events to digital or physical delivery consumers based on even or odd numbers, using Docker Compose and YAML configurations.

Explore dynamic routing that decides at runtime whether to route digital deliveries to a digital delivery topic or physical deliveries to FedEx or USPS topics based on carrier availability checks.

Set up a dynamic routing project with two consumers, FedEx and USPS, using topics FedEx delivery and USPS delivery, and a carrier availability service to switch bindings based on availability.

Demonstrates dynamic routing of order events with a producer, processor, and digital delivery consumer to FedEx and USPS using docker compose. See how availability shifts routing across partitions.

a quick recap on event routing, covering content-based and dynamic routing, with two Spring Cloud Stream options: stream bridge for consume-first routing, or header-based routing.

Learn how a Kafka cluster achieves high availability and horizontal scalability through brokers, partitions, and replication. Explore how the replication factor, partition leadership, and followers distribute load and ensure availability.

Understand how Kafka listeners separate control plane, data plane, and external access; brokers expose internal and external addresses via bootstrap servers to guide producers to the right broker.

Set up a three-node kafka cluster with docker compose by assigning unique node ids and a shared cluster id, and configure process roles with labeled internal, controller, and external listeners.

Configure a three-node Kafka cluster by examining internal and external listeners, advertised endpoints, node IDs, and controller quorum voters, while highlighting consumer offsets topic and disabling topic auto-creation.

Override broker-level kafka server properties using environment variables with the Kafka_ prefix, uppercase property names, and dots-to-underscores rules, useful for docker or kubernetes deployments.

Explore a three-node Kafka cluster defined in docker compose. Create three brokers named Kafka1, Kafka2, Kafka3 with common server.env properties, per-node variables, and port mappings 9092, 9093, 8081.

Inspect three-broker kafka cluster, create a topic with two partitions and replication factor three, and verify leaders and in-sync replicas while simulating broker failures to ensure controller election with majority.

Demonstrates high availability and fault tolerance in a three-node Kafka cluster managed by docker-compose, showing continuous message consumption despite bootstrap server failures.

Learn how topic partitions and replication enable scalability and availability in a Kafka cluster beyond broker capacity, and verify resilience through a Docker-based setup with bootstrap behavior.

Optimize producer side batching to boost throughput in Kafka with Spring Cloud Stream by tuning linger.ms, batch.size, and optional compression (LZ4) to batch and send messages efficiently via stream bridge.

Enable true end-to-end batching by processing list payloads on the consumer side with spring cloud stream, using list-based consumers to achieve batch delivery.

Set up a batch-oriented Spring Boot project using stream bridge to produce and consume messages, with batch mode and producer/consumer configs like linger and batch size.

Run end-to-end batch production and consumption using a producer and consumer in a Kafka setup, with docker-compose, section runner, and bulk processing of one million messages.

Explore batch processing in Spring Cloud Stream to extract message keys and payloads, using a single message object that provides lists of keys and payloads and how to iterate them.