Kafka Streams - Become a Streams Superhero

Learn how topology encapsulates the complete logic of Kafka Streams, guiding source-to-sink processing through steps like filter, split, join, and aggregation, with processor nodes forming a multi-way tree.

Create a topology by adding a topology folder and a basic topology class annotated for scanning; implement create topology with a stream builder for KStream or KTable and return build.

Explore the DSL and processor API in Kafka streams, showing when high-level manipulations suffice and when the low-level API offers flexibility. Learn KStream, KTable, and GlobalKTable.

Discover k stream, a core DSL abstraction in Kafka streams, handling immutable records with stateless operations like map values and filter and stateful joins and group by, plus multiple sinks.

Master KStream basics by consuming from single, multiple, and pattern topics, using default string serdes, then producing to new topics and validating results with Kafka UI.

Learn how KTable in the Kafka Streams DSL upserts the latest value per key, creating stateful data and enabling conversion to a key stream for topic publishing.

Master the global ktable, which loads the entire topic into one task for fast joins, is read-only, and can cause memory bottlenecks for large datasets.

Explore the processor in the Kafka Streams low level API, learning its three parts init, process, and close to manipulate, enrich, and aggregate records.

Participate in a hands-on session on the processor API for Kafka streams, building a source, processor, and sink topology and using process over the deprecated transform.

Explore how Kafka streams use the state store to perform stateful operations like aggregations and joins, using RocksDB as embedded storage with a change log for recovery.

Repartitioning in Kafka Streams routes identical keys to the same partition via an internal topic. Changing keys incurs latency and performance costs, with methods to adjust keys when needed.

Master serialization and deserialization in Kafka streams, converting objects to byte streams and back. Use standard serializers for basic types and custom serializers for objects like JSON, Avro, and Protobuf.

Build a custom serdes for kafka streams using a Pojo and json serialization. Implement a utils class with a generic getService method and wire it into a Ktable topology.

Understand how tasks and threads drive streaming work: partitions define tasks, threads execute them, and rebalance redistributes when a consumer crashes, with logs showing task formats like 0_1.

Explore how Kafka streams join K streams, K streams to K tables, and K tables to K tables with time windows and state store, covering left, inner, and outer joins.

Master co-partition by ensuring equal partitions, same partition strategy, and matching key types to join streams. When noncompliant, rekey or retopic, or consider a global KTable with memory caveats.

Explore KStream-to-KStream joins using a two-topic example of products and coupons, keyed by product ID, with windowing to align records within a time frame.

Join two streams by matching keys within a five-minute window to demonstrate KStream to KStream joins with inner, left, and outer variants, using DTO classes.

Apply the kstream to ktable join to enrich each product with the latest coupon data from the k table, using inner or left joins to handle missing keys.

Demonstrates a k stream to k table join, converting a k stream to a k table, removing windows, and comparing inner and left joins with trigger behavior.

Explore k table to k table joins to merge the latest product and coupon data into an integrated k table, triggering joins on updates and mirroring SQL joins with keys.

Explore k table to k table joins by converting streams to k tables, applying inner, left, and outer joins, and creating a materialized view to store join results.

Explore how k stream to global k table joins enrich streaming data with a read‑only global table, using a key value mapper to define the joining key and boost performance.

Enrich cart transactions by joining a k stream with a global key table using a key value mapper, demonstrating inner and left joins to produce checkout records with customer data.

Explore how Kafka Streams uses the DSL API to group data with a k stream or K table, and perform on-the-fly aggregations such as count, reduce, and aggregate.

Learn how to group KStream and KTable data in Kafka Streams, using group by and group by key, with repartition when changing keys and mandatory aggregation.

Execute a count aggregation in Kafka Streams to tally messages per key, producing a key table with long values and guiding further aggregation in the next lecture.

Explore the reduce method in Kafka Streams, using adder and subtractor reducers to aggregate key stream and key table data, such as calculating average stock prices while preserving value types.

Explore the aggregate method in Kafka Streams, compare it with reduce and the DSL, and learn how to use an initializer, aggregators, and a value service.

Review the variations of group by and group by key for k stream and k table, including key selector and key value mapper, with default and specific windows.

Build a Kafka Streams topology to group by key and by key-value mappings, apply reduce to aggregates, and publish grouped stock data to new topics.

Explore grouping with ktable in kafka streams through a hands-on session, converting kstreams to ktables and handling the key value mapper for group by and serialization concerns.

Explore the count methods in Kafka Streams, distinguishing k group stream and k group table variants, including variants with processor naming and materialized views, all returning long counts per key.

Learn to use count methods on a k stream, with group by key, and materialized views, naming processors, and printing per-key counts using a pick lambda in a hands-on demo.

Clone count topology, switch from k stream to k table, and apply group by and key value mapper to perform counts. Compare parameterless and named processor variants with materialized views.