Kafka Streams with Spring Cloud Stream

Learn to install and run Confluent Kafka on Windows using the Windows Subsystem for Linux (WSL), configure brokers, and run a local development cluster with topics, producers, and consumers.

Create your first Kafka Streams project on Windows with Spring Cloud Stream, binding an input channel to the users topic and logging each received message.

Install and run confluent kafka on mac, configure broker settings, and use the confluent control center to create topics and test producers and consumers.

Create your first Kafka Streams application using Spring Cloud Stream, bind an input channel to the users topic, and log each message with a Kafka Streams binder.

Understand spring's two kafka paths: spring for apache kafka for producer/consumer APIs, and spring cloud stream with the kafka streams binder for kstream and ktable processing.

Explore how Spring Cloud Stream enables event-driven microservices with input/output bindings and binders, including the Kafka Streams binder.

Discover how Spring Cloud Streams connects to Kafka and uses Kafka Streams DSL to transform data from KStream and KTable with aggregates, joins, and windowing in a microservice.

Build a Spring Boot microservice that exposes a post API to receive JSON with topic, key, and value and publish to a Kafka topic using a KafkaTemplate from Spring Kafka.

Design a retail pos simulator that generates realistic invoices (home-delivery or takeaway) and publishes them to a Kafka topic via an invoice generator and a Kafka producer service, highlighting serialization.

Produce json invoices and publish them to a Kafka topic using Spring Boot, with address, line item, and invoice generators, a Kafka producer service, and json serialization.

Produce avro serialized invoices to Kafka by generating avro friendly classes from schemas with the avro maven plugin and configuring the avro serializer with Confluent Schema Registry.

Develop a Kafka streams application that reads invoices from a pos-topic, sends home-delivery invoices to the shipment service, and primes customers for loyalty notifications, while masking data for Hadoop analytics.

Develop a Kafka Streams pipeline with Spring Cloud Stream, define JSON input and Avro output models, implement three transformation functions, and process invoices into notifications and Hadoop records.

Explore real-life serialization scenarios in Kafka Streams with Spring Cloud Stream, processing JSON invoices to Avro or Avro to JSON using Schema Registry and Confluent tools.

Read Avro formatted invoices from a Kafka topic, transform them into notification and HadoopRecord, and publish json messages to loyalty and Hadoop-sink topics using Spring Cloud Stream.

Explore how the Kafka streams binder handles serialization and deserialization of byte-arrays with Serde. Learn why native encoding and decoding by Kafka Streams is recommended for input and output channels.

Understand the KStream object in Kafka Streams, its functional style, and core methods like filter, mapValues, and foreach, plus peek as a non-terminal alternative.

Learn how to implement exactly once transactions in Kafka Streams by reading input once and producing to multiple topics, achieving atomic commits and avoiding duplicates in failure scenarios.

Implement exactly-once transactions in a single listener by setting processing.guarantee to exactly-once, using avro serde, and routing kstream outputs to topics with to().

Build a Kafka Streams app with Spring Cloud Stream that ingests XML orders, converts to JSON, validates, and routes to india-orders or abroad-orders with error handling and exactly-once processing.

Learn to read XML inputs in a Kafka Streams with Spring Cloud Stream project by mapping XML to Java objects using JAXB and the JAXB2 plugin, with Java 8 compatibility.

Explore building a real-time streaming app with Kafka Streams and Spring Cloud Stream, validating orders with mapValues and tagging and routing them via branch to valid or error topics.

Explore mixed branching of a kstream by parsing xml orders into an OrderEnvelop, routing invalid orders to an error topic, and splitting valid orders to india-orders and abroad-orders.

Learn to handle poison pills in Kafka Streams with Spring Cloud Stream by choosing logAndFail, logAndContinue, or sendToDlq, and configure a deserializationExceptionHandler and DLQ topics per input channel.

Explore how the KTable provides a database-like upsert model with a primary key constraint, where null values delete and updates overwrite records; learn to create, transform, and aggregate KTable streams.

Explore how to build a simple Kafka streams app using KTable to read stock-ticks, compare KTable and KStream behavior, apply filter, and convert back to KStream for output.

Demonstrate real-time streaming aggregates by building a word count app that reads from a topic, splits lines into words, counts them with KTable, and prints updates.

Learn how to compute aggregates in Kafka Streams by grouping data by key and applying reduce or aggregate formulas, with attention to KStream versus KTable and repartitioning.

Learn how to implement cumulative loyalty points using the Kafka Streams reduce() method, transforming PosInvoice data into Notification messages with total loyalty points across invoices.

Aggregate a Kafka stream with the aggregate method to compute department salary aggregates from Avro employee records, grouped by department, using initializer and aggregator lambdas in real time.

Examine real-time aggregation challenges, including negative aggregation, using rewards and department salary averages. Learn why KStream treats records as additions and when to use KTable for update streams.

Explore KTable aggregation in Kafka Streams with Spring Cloud Stream, using group by department and aggregate to update department averages as employees move.

Explore how Kafka time semantics drive windowed processing and joins by configuring event time, ingestion time, or processing time using timestamp extractors.

Group invoices by store id and count them in five-minute time windows using event-time semantics with a timestamp extractor, implemented in a Spring Cloud Stream Kafka Streams app.

Explore tumbling and hopping time windows in Kafka streams, contrasting fixed-size, non-overlapping windows with overlapping windows defined by size and advance interval for moving aggregates.

Define a session as activity with a five-minute idle gap. Group clicks by user id, apply a five-minute session window, and count clicks per session using event-time semantics.

Learn how to join KStream, KTable, and GlobalKTable in Kafka Streams, using inner, left outer, right outer (swap), and full outer joins, with co-partitioning and key requirements.

Explore KStream to KStream joins as windowed operations using a five-minute JoinWindow, uniting payment request and confirmation streams to validate OTPs in real time.

Learn to implement KTable to KTable joins in Kafka Streams with Spring Cloud Stream, producing a KTable and updating last login timestamp while ensuring co-partitioned data with the same key.

Learn to join ad clicks with ad inventories using a KStream to KTable join and GlobalKTable enrichment to count clicks by news type in real time.

Learn to build complex aggregations in Kafka Streams by combining joins and counts to derive the top 3 news types by clicks, using a custom top-3 sorter and stateful aggregation.