The Ultimate Kafka Streams (3.x) : Real-time Data Processing

Explore Kafka streams, a Java library for enriching, transforming, and processing real-time data in the Kafka ecosystem. Use its DSL for streams and tables, with stateless, stateful processing and windowing.

Explore key terms in Kafka Streams, including dataflow programming, DAG topology, source processors, stream processors, sink processors, and tasks for parallel processing.

Explore the Kafka Streams parallel mode and deployment approaches: single server with multiple threads, multiple instances on a single server, and multi-server deployments delivering high performance and availability.

Explore data process strategy in Kafka Streams, highlighting depth-first processing of records through a topology, buffered by the source processor and shaped by topic partitions for parallel processing.

Set up a Kafka broker on Linux, configure ZooKeeper, broker properties, and test with a topic. Next, set up development tools and start the first Kafka Streams app.

Create a Maven Kafka Streams project in IntelliJ, add dependencies and log4j, and implement topology that reads from source topic, applies filter, maps to uppercase, and writes to target topic.

Define Kafka streams config, build a topology with StreamsBuilder, apply filter and map transformations, publish results to the target topic with proper serdes, and ensure graceful shutdown.

Kick off the first Kafka Streams application, verify real-time processing by producing from input.words and consuming from output.words, applying a length-based filter and an uppercase transformation.

Explore stateless streams in Kafka Streams, where each event is processed independently with no memory of prior events, embodying stateless operation and isolation.

Learn how the map operation in Kafka streams transforms each input record into a new output by changing key and value types, returning a new stream and potentially repartitioning.

Map values operation affects only the value, keeping the key unchanged and avoiding internal redistribution; use ValueMapper or ValueMapperWithKey to transform values and create a new kstream.

Use filter and filterNot on Kafka streams to keep or drop records by a predicate, producing a new kstream while preserving the key and avoiding repartition.

Explore flat map in Kafka streams, transforming each input record into zero or more outputs via KeyValueMapper returning an Iterable, producing a new kstream without terminating the current stream.

Learn how flatMapValues uses the ValueMapper to produce multiple outputs from a value while keeping the key unchanged, including value mapper with key, an intermediate stateless operation with no repartitioning.

Discover how the selectKey operation uses a KeyValueMapper to assign a new key for each input record. This stateless, record-by-record transformation enables downstream grouping, joining, and aggregation.

Learn the stateless foreach operation in Kafka streams, a terminal action that logs each record and returns void, and sink results to the target topic.

Explore two stateless ops, print and peek, for debugging Kafka Streams data processing; print outputs to console, but not recommended for production due to performance, while peek preserves records.

Explore stateless split operations in Kafka streams, routing records to BranchedKStream by predicates and creating substreams named Apache, Kafka, and Streams with a default branch.

Merge smaller kstreams into a bigger one using a stateless operation in Kafka Streams, requiring identical keys and value types, and a named merge processor.

Sink transformed records to a target topic in a stateless operation for downstream consumers like spark streaming or kafka sink connect. Configure key and value serdes to avoid serialization errors.

Assemble a complex Kafka Streams application to analyze XMall real-time transaction data, masking credit cards, extracting zip codes, deriving transaction keys and patterns, computing rewards, and routing to multiple topics.

Define and implement the transaction data model for real-time analysis with Kafka Streams, including Transaction, TransactionKey, TransactionPattern, and CustomerReward, using a builder pattern and credit card masking.

Define json serdes for the transaction data model, including Transaction, TransactionKey, TransactionPattern, and CustomReward, by implementing serializer and deserializer, configuring ObjectMapper with JavaTimeModule, and exposing TransactionSerde for real-time processing.

Build a six-topic real-time XMall transaction workflow that consumes raw data, masks PII, extracts patterns, computes rewards, filters purchases over 5 USD, and branches coffee and electronic to dedicated topics.

Utilize kafka streams to deploy and test the xmall transaction real-time analysis, creating topics, starting the application, and verifying pattern, rewards, purchase, coffee and electronic transactions in real time.

Recap of Kafka stateless streams and operations like map, mapValues, filter, flatMap, selectKey, forEach, plus debugging with print and peek and sink to materialize results for real-time transaction analysis.

Explore stateful streams in Kafka, contrasting stateless versus stateful operations, and learn how capturing and remembering events and behavior enables joins and aggregations.

State stores hold remembered data for stateful operators in a Kafka Streams application, with embedded RocksDB at the task level and changelog-backed recovery for fault-tolerant operation.

Implement a Kafka Streams stateful transformation with a statestore to perform real-time word count, using a stateful transform-processor and a target topic.

Demonstrate stateful transformation with a word count application using a statestore in Kafka Streams, producing and verifying results on source and target topics.

Explore how Kafka streams redistributes data for correct stateful transforms, using repartition topics and local state stores synchronized to change logs, and restart procedures to fix wrong results.

Enhance the XMall real-time transaction processing with a stateful reward point processor that tracks both current and total rewards using a state store and a value transformer in Kafka Streams.

Master stateful transform operations in kafka streams, including transform and flat transform with key supplier and iterable outputs. Upstream key changes won't auto-repartition; call repartition explicitly.

Learn Kafka Streams join operations, a stateful method to combine two streams by key within a time window, buffering with a state store, featuring inner, left outer, and outer joins.

Master inner join on KStream by aligning keys and timestamps within a join window, using ValueJoiner to produce results while ignoring null keys or values.

Explore kstream inner joining by producing records to user.info and user.address, showing that only records with the same key and in the same time window emit to the output.

Explore the inner workings of Kafka Streams' inner join, including key selection with map, automatic data redistribution, and the creation of state stores and their change logs for one-minute windows.

Explore the KStream left-joining operation in Kafka Streams, contrasting it with inner join, showing left join emits records with null address until a match arrives within the same time window.

learn the KStream outer joining operation, a full join where all records from both streams are produced at least once within a window, with a practical code demo.

Explore the grouping and aggregation operation as the basis for stateful processing in Kafka streams, including redistribution to align keys with partitions and the new KGroupdStream abstraction.

Explore the count aggregation on a KGroupedStream, counting records by the grouped key and writing results to a materialized local KeyValueStore named 'word-count'.

Demonstrate the KGroupedStream reduce aggregation operation by implementing a word-count app with a reducer, named processor, and materialized state store, using lambda and static infer.

Implement a real-time sales champion app using reduced aggregate operations in kafka streams, with per-salesman totals, department grouping, and department champions displayed on a dashboard.

Develop a real-time sales champion analysis with Kafka Streams by building a topology that maps values, aggregates sales by salesman and department, and outputs the department champion in real time.

Transform the sales data into SalesStats using the aggregate operation to compute per-department total and average amounts in real time.

Expose a restful, queryable statestore API to access Kafka Streams state via a read-only key-value store, enabling interactive queries of stateful data.

Deploy a kafka streams app to multiple instances by parameterizing the statestore directory and port, configuring streams threads below partitions, packaging with the assembly plugin, and running separate instances.

Explains the stateful process operation in Kafka streams, accesses the statestore, and shows external sinks, but recommends emitting to a target topic and using a Kafka Connect sink, noting bugs.

Recap stateful streams with embedded RocksDB statestore, explore storage options, and build a word count stateful Kafka Streams app, covering repartition, change logs, transforms, joins, grouping, and aggregations like count.

Explore Kafka streams time semantics for windowed operations, comparing event time, ingestion time, and processing time, to select accurate timestamps for windowed joins and aggregations.

Understand Kafka timestamp semantics with CreateTime and LogAppendTime, configure topic and broker settings, and observe event time versus ingestion time in producer and consumer windowing operations.

The Kafka Streams framework provides a timestamp extractor to assign a timestamp to each record for joins and aggregations. The TimestampExtractor interface exposes an extract method returning long, with FailOnInvalidTimestamp.

Explore the Kafka streams windowing operation, grouping by key, sub-grouping by time into tumbling, hopping, sliding, and session windows, with per-window state stores and retention.

Use tumbling time window aggregation on Kafka streams to detect real-time network attacks by grouping logs by remote IP in fixed five-minute windows and flag IPs with over 10 accesses.

Leverage Kafka Streams to perform real-time network attack behavior analysis using a tumbling one-minute window, grouping by remote address and counting events to trigger warnings when hits exceed ten.

Kafka Streams continuously emits intermediate window results as data arrives, but you can suppress updates and emit only the final window results downstream when the window closes using Suppressed.untilWindowCloses.

Learn how hopping time windows enable real-time website traffic statistics with fixed-sized, overlapping intervals, a 1-minute window, and 10-second hops.

Explore sliding time windows for real-time website access traffic, where output fires only when the window content changes, offering more efficient processing than hopping windows.

Explore session time windows and sessionization to count user visits within inactivity gaps in real-time statistics, using session windows with a 5-minute gap.

Assemble a real-time health monitoring pipeline using kstreams and ktables to join heartbeat events with patient and sick room data, alerting doctors when heart rates exceed normal ranges.

Implement real-time health monitoring by streaming heartbeat data with a one-minute tumbling window. Join patient and sickroom data to produce PatientWithSickRoom and issue health check monitor warnings to doctors.

Produce patient and sickroom records and stream heartbeat data with Kafka, join kstream to ktable, and send doctor monitoring warnings for age and heartbeat thresholds.

Learn to query a Kafka Streams window store via the interactive query API in a KGroupedStream setup, exposing a heartbeat endpoint that returns windowed heartbeat data in JSON.

Explore Kafka Streams windowing on KGroupedStream, covering event time, ingestion time, and processing time, with timestamp extractor. Apply tumbling, hopping, sliding, and session windows to heartbeat sensor data.

Understand the KTable as a changelog-based upsert stream where each record updates a key's last value, with nulls as deletes, illustrated by Alice's updates and stock prices.

Learn how to create a KTable from StreamsBuilder using table(), toTable(), and windowed aggregates, with materialized state stores and changelog implications.

Create a KTable from a KStream by transforming the stream to add a key, then apply toTable with a materialized users-state-store and string serdes.

Explore KTable basics, including filter and mapValues operations, and tombstone semantics where null values delete records; convert to KStream to enable print, forEach, and sink.

Demonstrates KTable basic operations and tombstone semantics, using salary-based title evaluation and filter and mapValues, with tombstones propagating to downstream topics.

Leverage KTable transformValues to create a statestore backed by a key value store, handling tombstone records and streaming updated ShootStats for real-time player score tracking in the shoot game.

Perform a non-windowed KStream inner join with a KTable using a primary key lookup, emitting results only when stream records match, while KTable updates update state without new outputs.

Demonstrates how a KStream left join KTable outputs the joined result even when the address is null.

Learn how KTable inner joins with another KTable to produce joined results from user.info and user.address using a value joiner, while handling tombstones and null deletes.

Explore KTable inner joins in Kafka Streams (3.x) by using a foreign key extractor to join a left KTable's value-part with a right KTable, illustrating real-time data processing.

Explore the KTable API's joining operations in Kafka Streams, including inner and left joins with KTable and foreign key joins on another KTable, with clear, actionable patterns.

Explore KTable grouping and count aggregations to compute employees per department using KGroupedTable and repartitioning; demonstrate updates, deletions, and tombstone records.

See how ktable reduce aggregates update department salary totals in real time, with adder and subtractor reducers, to compute total salary per department and emit downstream.

Refactor a total salary per department using KTable grouping and aggregate to produce EmployeeStats with department and total salary, using initializer and subtractor, and materialize as aggregate with value serde.

Master KTable concepts, including its update of a KStream and last-key retention, explore creation approaches, tombstone records, and core operations like count and aggregate.

Explore GlobalKTable as a changelog-stream abstraction for a primary-keyed table, enabling key lookups and right-hand side stream-table joins in Kafka Streams DSL, with inner and left join limitations.

Compare globalKTable and KTable, showing how globalKTable broadcasts all partitions to every instance for efficient joins and small static data, with code using globalTable().

Demonstrate a KStream inner join with a GlobalKTable to enrich user.info using a KeyValueMapper and a ValueJoiner. This right-hand join creates a joined stream, labeled stream-join-globalKTable, with no time window.