The Ultimate Kafka Streams (3.x) : Real-time Data Processing

Name: The Ultimate Kafka Streams (3.x) : Real-time Data Processing
Rating: 4.7 (55 reviews)

Apache Kafka Streams: Stateless Streams, Stateful Streams, KTable, Window, Statestore, RocksDB,Real-Time data analysis

Highest Rated

Created byAlexander Wong

Last updated 10/2022

English

What you'll learn

Fullly understand Real-Time data process model of Kafka Streams
In Depth understand Kafka Streams stateless operation
In Depth understand Kafka Streams stateful operation
In Depth understand Kafka Streams KTable & GlobalKTable
Build Complex Event Process Application
Rocksdb and statestore in Kafka Streams

Course content

7 sections • 87 lectures • 7h 31m total length

Course content introduction3:27
Explore real-time data processing with Kafka streams 3.0.0, covering stateless and stateful operations, KStream, KTable, windowing, and heart rate sensor data health monitoring.
Course objective and scope1:39
Master stateless and stateful Kafka Streams applications, real-time streaming with the Kafka Streams framework, and complex event processing, while covering prerequisites in Java 8, Kafka fundamentals, and Maven-built Java projects.
Course Characteristics & how to learning (suggestions)1:37
Explore the ultimate Kafka Streams (3.x) real-time data processing course, featuring step-by-step knowledge points, source code exercises, and real-case applications with diagrams to illustrate underlying concepts.
Download the source code & PowerPoint0:02

What is Kafka Streams4:44
Kafka Streams Key Terms5:38
Kafka Streams Parallel mode2:40
Data process strategy in Kafka Streams2:21
Explore data process strategy in Kafka Streams, highlighting depth-first processing of records through a topology, buffered by the source processor and shaped by topic partitions for parallel processing.
Kafka Streams Development Environment Setup(Kafka Broker Setup)7:36
Kafka Streams Development Environment setup(Maven Project)4:46
Quick Start-Develop First Kafka Streams Application9:49
Setup the First Kafka Stream Application and Testing2:24

What is stateless streams application1:43
Stateless Operation- Map & KeyValueMapper5:36
Learn how the map operation in Kafka streams transforms each input record into a new output by changing key and value types, returning a new stream and potentially repartitioning.
Stateless Operation- MapValues & ValueMapper & ValueMapperWithKey5:05
Stateless Operation- Filter & FilterNot & Predicate4:37
Stateless Operation-FlatMap & KeyValueMapper4:32
Stateless Operation-FlatMapValues&ValueMapper&ValueMapperWithKey4:05
Stateless Operation-SelectKey & KeyValueMapper3:59
Discover how the selectKey operation uses a KeyValueMapper to assign a new key for each input record. This stateless, record-by-record transformation enables downstream grouping, joining, and aggregation.
Stateless Operation-Foreach & ForeachAction3:16
Stateless Operation-Print and Peek2:41
Explore two stateless ops, print and peek, for debugging Kafka Streams data processing; print outputs to console, but not recommended for production due to performance, while peek preserves records.
Stateless Operation-(BranchedKStream)Split Streams to multiple sub-streams7:59
Stateless Operation-Merge Small KStreams to big one2:30
Stateless Operation-Sink the transformed records to target topic4:32
Assemble Together: XMall Transaction data real-time analysis practise5:07
Assemble Together: XMall Transaction data real-time analysis - Data Model Define9:36
Define and implement the transaction data model for real-time analysis with Kafka Streams, including Transaction, TransactionKey, TransactionPattern, and CustomerReward, using a builder pattern and credit card masking.
Assemble Together: XMall Transaction data real-time analysis-Custom Serdes7:04
Assemble Together: XMall Transaction data real-time analysis-Streams Application12:01
Assemble Together: XMall Transaction data real-time analysis-Deploy & Testing5:12
Kafka Stateless Streams Recap & Summarize1:57

What is stateful streams application2:57
State store & state store solution4:51
Stateful transformation practical (Word Count)9:50
Stateful transformation practical(Word Count) test and result verify3:35
Understand the Kafka streams internal data redistribution and stateful transform10:26
Explore how Kafka streams redistributes data for correct stateful transforms, using repartition topics and local state stores synchronized to change logs, and restart procedures to fix wrong results.
Enhancement XMall transaction reward point processor support total reward points9:19
Enhance the XMall real-time transaction processing with a stateful reward point processor that tracks both current and total rewards using a state store and a value transformer in Kafka Streams.
Stateful transform operations2:11
Master stateful transform operations in kafka streams, including transform and flat transform with key supplier and iterable outputs. Upstream key changes won't auto-repartition; call repartition explicitly.
KStream Joining operation introduction2:52
KStream inner joining operation9:23
Master inner join on KStream by aligning keys and timestamps within a join window, using ValueJoiner to produce results while ignoring null keys or values.
KStream inner joining operation application startup & testing3:04
Depth understand the joining operation and underlying change logs4:32
KStream left joining operation3:13
KStream outer joining operation2:17
learn the KStream outer joining operation, a full join where all records from both streams are produced at least once within a window, with a practical code demo.
KStream grouping operation2:16
KGroupedStream count aggregation operation practical(word count)6:38
KGroupedStream reduce aggregation operation practical(word count)5:40
Real-Time analysis the sales champion application by reduce aggregate operation2:51
Implement a real-time sales champion app using reduced aggregate operations in kafka streams, with per-salesman totals, department grouping, and department champions displayed on a dashboard.
Develop the sales champion application by reduce aggregating operation and test7:58
KGroupedStream aggregate operation for Real-Time compute sales stats reporter11:33
Stateful KStream Queryable Storestore(KeyValueStore)-single instance9:49
Stateful KStream Queryable Storestore(KeyValueStore)-multiple instance6:15
Deploy a kafka streams app to multiple instances by parameterizing the statestore directory and port, configuring streams threads below partitions, packaging with the assembly plugin, and running separate instances.
Stateful process and ProcessorSupplier(Official documentation has bugs)3:19
Kafka Stateful Streams Recap & Summarize1:57
Recap stateful streams with embedded RocksDB statestore, explore storage options, and build a word count stateful Kafka Streams app, covering repartition, change logs, transforms, joins, grouping, and aggregations like count.

Kafka Streams Time Semantics3:46
Explore Kafka streams time semantics for windowed operations, comparing event time, ingestion time, and processing time, to select accurate timestamps for windowed joins and aggregations.
Kafka LogAppendTime&CreateTime5:26
Kafka Streams TimestampExtractor3:06
The Kafka Streams Windowing Operation introduction4:11
Tumbling time window analyzes network attack behavior in real time4:16
Tumbling time window real-time analysis of network attack behavior app building8:02
Leverage Kafka Streams to perform real-time network attack behavior analysis using a tumbling one-minute window, grouping by remote address and counting events to trigger warnings when hits exceed ten.
Suppress some updates from this changelog stream4:36
Hopping time window for real-time statistics the website access traffic8:06
Sliding time window for real-time statistics the website access traffic8:47
Hopping time window vs Sliding time window0:34
Session time window for real-time statistics website PV&UV6:03
Explore session time windows and sessionization to count user visits within inactivity gaps in real-time statistics, using session windows with a 5-minute gap.
Assembly-Heartbeat sensor data real-time analysis patient health monitoring-I2:55
Assembly-Heartbeat sensor data real-time analysis patient health monitoring-II14:01
Assembly-Heartbeat sensor data real-time analysis patient health monitoring-III4:35
KGroupedStream Interactive Queryable Storestore(WindowStore)5:47
Learn to query a Kafka Streams window store via the interactive query API in a KGroupedStream setup, exposing a heartbeat endpoint that returns windowed heartbeat data in JSON.
KGroupedStream & Windowing Aggregation Recap and Summarize1:39

What is KTable2:52
Create KTable from StreamsBuilder(Official documentation has bugs)10:31
Create KTable from KStream4:00
KTable basic operation introduction and Tombstone records explanation2:43
Explore KTable basics, including filter and mapValues operations, and tombstone semantics where null values delete records; convert to KStream to enable print, forEach, and sink.
Lab: KTable basic operation exercises(contains tombstone records)9:45
KTable transformValues operation and shoot game15:56
KStream inner join the KTable6:27
KStream left join the KTable2:29
KTable inner join the KTable3:38
Learn how KTable inner joins with another KTable to produce joined results from user.info and user.address using a value joiner, while handling tombstones and null deletes.
KTable Foreign Key inner join the KTable4:27
Explore KTable inner joins in Kafka Streams (3.x) by using a foreign key extractor to join a left KTable's value-part with a right KTable, illustrating real-time data processing.
KTable supported joining operations1:05
KTable grouping & count aggregating count the number of employees6:21
Explore KTable grouping and count aggregations to compute employees per department using KGroupedTable and repartitioning; demonstrate updates, deletions, and tombstone records.
KTable grouping & reduce aggregating statistics department total salary9:16
See how ktable reduce aggregates update department salary totals in real time, with adder and subtractor reducers, to compute total salary per department and emit downstream.
KTable grouping & aggregate aggregating statistics department total salary7:36
KTable Recap and Summarize1:17

Requirements

Java(1.8+) Development Experiences
Apache Kafka Fundamental Knowledge
Big Data Development Experiences
Apache Maven project build tools

Description

**** Please enable the vedio cc function (captions ) *****

First of all, welcome to enroll this course. This is a course about Kafka Streams. In this course, every knowledge detail of the Kafka Streams framework is introduced in great detail. Secondly, I sincerely hope that you can enable the vedio cc function (captions ) , because my native language is not English, the spoken language is not very standard, but I assure you that the course content is absolutely detailed and step by step，From shallow to deep.

Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka clusters. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster technology.

[Pre-Requisites]

You should have the Java development experiences(***this is mandatory requirement***)
You should have the Kafka foundation knowledge(***this is mandatory requirement***)
It's better have another streaming develop experiences such as Spark Streaming, Storm, Flink

【Course Characteristics】

Driven by source code
Lots of practices
From shallow to deep
Absolutely detailed and step by step
Covers all knowledge points of Kafka Streams framework
Rich comprehensive cases

[Course Agenda]

Introduce the Kafka Streams
Tutorial the Kafka Streams key terms and concepts
Kafka Streams Parallel Mode
Stateless operation of map transform
Stateless operation of mapValues transform
Stateless operation of flatMap transform
Stateless operation of flatMapValues transform
Stateless operation of selectKey transform
Stateless operation of foreach
Stateless operation of Print&Peek
Stateless operation split & merge & BranchedKStream
How to custom Serdes
XMall Transaction data real-time analysis practise
Tutorial the kafka stateful operation and statestore
Explain in details of internal data redistribution and stateful transform
Stateful operation of Joining(inner join/left join/outer join)
Stateful operation of grouping
Stateful operation of aggregation(count,reduce,aggregate)
Build Real-time analysis the sales champion application
Build Real-time analysis the sales stats application
Stateful KStream Queryable Storestore
Stateful TimeWindowedKStream Queryable state store for interactive
KGroupedStream windowing operation
Time Semantics and custom TimestampExtractor
Tumbling time window for analysis of Potential Cyber Attacks
Hopping time window for Site Visit real-time statistics
Heartbeat sensor data real-time analysis for patient health monitoring
What is KTable and how to create the KTable
KTable basis operation such as map values, filtering
KTable basis stateful operation transformValues implement the shooting game
KStream inner&left join the KTable enrichment/enhancement the orginal records
KTable inner join, inner foreign key with other KTable
KTable left join, left foreign join, outer join KTable
KTable & KGroupedTable aggregating operation such as count/reduce/aggregate

[Course Objectives]

Fully understand the kafka Streams concepts and key terms
Fully understand the kafak Streams parallel mode
Master the stateless streams application building and in depth understand every stateless operation
Master the stateful streams application building and in depth understand every stateful operation
Master the internal data distribution underlying mechanism
Master the statestore, can base on the statestore build complex event process real-time application
Fully understand the KTable and Windowing operation

Hope you will enjoy this course, After learning this course, you will become an expert in Kafka Streams, and ability to build complex event process(CEP) real-time application based on Kafka Streams framework.

Who this course is for:

Big Data Developer
Senior Java/Scala/Groovy/Clojure Developer
Kafka Engineer
Big Data Engineer

The Ultimate Kafka Streams (3.x) : Real-time Data Processing

What you'll learn

Explore related topics

Course content

Course Introduction4 lectures • 7min

Apache Kafka Streams Introduction8 lectures • 40min

Build Kafka Stateless Streams Application18 lectures • 1hr 32min

Build Kafka Stateful Streams application & statestore(RocksDB)23 lectures • 2hr 7min

Build complex streaming application with KGroupedStream windowing operation16 lectures • 1hr 26min

Build Updatable Streams Application with KTable15 lectures • 1hr 28min

GlobalKTable API3 lectures • 12min

Requirements

Description

Who this course is for: