Apache Kafka Series - Kafka Streams for Data Processing
4.5 (2,621 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
21,172 students enrolled

Apache Kafka Series - Kafka Streams for Data Processing

Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8
4.5 (2,621 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
21,172 students enrolled
Last updated 8/2020
Italian [Auto], Portuguese [Auto], 1 more
  • Spanish [Auto]
Current price: $90.99 Original price: $129.99 Discount: 30% off
23 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 5 hours on-demand video
  • 5 articles
  • 1 downloadable resource
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Write four Kafka Streams application in Java 8
  • Configure Kafka Streams to use Exactly Once Semantics
  • Scale Kafka Streams applications
  • Program with the High Level DSL of Kafka Streams
  • Build and package your application
  • Write tests for your Kafka Streams Topology
  • And so much more!
  • Good knowledge of Apache Kafka
  • A recent computer (Windows / Linux / Mac OS X) with 4GB of RAM
  • Good knowledge of Java 8 Programming Language

The new volume in the Apache Kafka Series!

Learn the Kafka Streams data processing library, for Apache Kafka. Join hundreds of knowledge savvy students into learning one of the most promising data processing library on Apache Kafka.

Kafka Streams
is the easiest way to write your applications on top of Kafka:

> Easiest way to transform your data using the High Level DSL
> Exactly Once semantics support out of the box!
> Deploy and Scale your Kafka Streams application without a cluster!
> Perform Aggregations, joins, and any operations you may think of using only a few lines of code!
> Built on top of Kafka, for fault tolerance, scalability and resiliency


Stephane Maarek is the instructor of this course. He is a Kafka Expert, and the author of the highly-rated Apache Kafka Series on Udemy, having taught already to 40000+ students and received 12000+ reviews.

He shares all his Kafka knowledge on the platform, taking the time to explain every concept and provide students with both theoretical and practical dimensions. You are in good hands!

Hands On & Theory Based Course

Note: This course is based on Java 8, and will include one example in Scala. Kafka Streams is Java based, and therefore is not suited for any other programming language. 

Each section can be either theoretical, or a practice section. 

> Through practice, you will be challenged by writing your own Kafka Streams application. The solutions will be thoroughly explained, and you will learn some tips on how to use Kafka Streams the best way.

> Through theory, you will learn about all the available API, the inner workings of the library, as well as some exciting concepts such as Exactly Once Semantics!

This course is the first and only available Kafka Streams course on the web. Get it now to become a Kafka expert!

Section outline:

  1. Kafka Streams - First Look: Let's get Kafka started and run your first Kafka Streams application, WordCount

  2. End-to-End Kafka Streams Application : Write the code for the WordCount, bring in the dependencies, build and package your application, and learn how to scale it. This is a complete end to end example

  3. KStream and KTable Simple Operations: Learn all the stateless operations available for the KStream and KTable API

  4. Practice Exercise - Favourite Colour: Practice your newly acquired skills by writing your own Kafka Streams application, Favourite Colour. It will be challenging! Includes a Scala version of the example

  5. KStream and KTable Advanced Operations: Learn all the stateful operations available for the KStream and KTable API

  6. Exactly Once Semantics - Theory: Learn what EOS (Exactly Once Semantics) is, how Kafka 0.11 enables it, and how to activate them in Kafka Streams

  7. Exactly Once - Practice Exercise - Bank Balance: Practice your newly acquired knowledge by writing your own Kafka Streams Exactly Once application, to compute a running Bank Balance for your customers

  8. Testing your Kafka Streams Application: Learn how to test the WordCount Kafka Streams Topology with Kafka Streams v1.1.0

This Course Also Comes With:

✔ Lifetime Access to All Future Updates

✔ A responsive instructor in the Q&A Section

✔ Links to interesting articles, and lots of good code to base your next applications onto

✔ Udemy Certificate of Completion Ready for Download

✔ A 30 Day "No Questions Asked" Money Back Guarantee!

Learn The Hottest Kafka Data Processing Library Now!

Kafka is increasingly becoming a must-have skill, and this course will set you up for fast success using the Kafka Streams API. The job market will need people with your newly acquired skillset!

"Take This Course" risk free and learn Kafka Streams now!

Who is the target audience?

  • This course is for developers and devops who would like to learn how to write, package, deploy and run a Kafka Streams applications

  • This course is for architects who would like to understand how Kafka Streams work and its position in the Kafka-centered data pipeline and enterprise architecture

  • You will need some good understanding of Kafka before starting this course

Who this course is for:
  • Developers and devops who would like to learn how to write, package, deploy and run a Kafka Streams applications
  • Architects who would like to understand how Kafka Streams work and its position in the Kafka-centered data pipeline and enterprise architecture
  • People who fully understand the basics of Kafka
  • Java 8 and Scala developers only
Course content
Expand all 76 lectures 04:49:45
+ Kafka Streams - First Look
5 lectures 27:09

Learn what is Kafka Streams at a high level

Preview 04:46

Run through some pre-requisites for this course, understand who the target student is

Preview 04:11

Short bio of your instructor, Stephane Maarek!

About your Instructor: Stephane Maarek

Full end to end run of your first Kafka Streams application.

We will download Kafka, start our own cluster, and run producers, consumers, and our first Kafka Streams application

Preview 12:52

Short lecture comparing Kafka Streams to other streaming libraries

Preview 03:00
+ End to End Kafka Streams Application - Word Count
15 lectures 01:12:42

Description of the Word Count Kafka Streams application

Section Objective

Learn the Kafka Streams Core Concepts, Topologies, Sink, Sources, Processors, Streams

Kafka Streams Core Concepts

Setup Java 8, Maven, IntelliJ IDEA Community Editions so we can get started coding our first Kafka Streams application

Environment and IDE Setup: Java 8, Maven, IntelliJ IDEA

Setup a project starter so we can include the right kafka streams dependencies and get coding!

Starter Project Setup

Write our first Kafka Streams code. We will setup the Kafka Streams properties

Kafka Streams Application Properties

Learn about what is a Java 8 Lambda and how we will use them

Java 8 Lambda Functions - quick overview

Write our application topology using the newly acquired Java 8 Lambdas.

This is the first time we're dealing with the Kafka Streams API

Preview 14:07

Learning how to print the Kafka Streams topology

Printing the Kafka Streams Topology

Add some code so that our Kafka Streams application shuts down gracefully when we stop it

Kafka Streams Graceful Shutdown

Learn how to run our Kafka Streams application straight from IntelliJ

Running Application from IntelliJ IDEA

Learn how to run the IntelliJ debugger to do step through in our code

Debugging Application from IntelliJ IDEA

Learn about Kafka Streams internal topics (repartition and changelog topics)

Internal Topics for our Kafka Streams Application

Learn how to use Maven Assembly to package our Kafka Streams application as a fat jar

Packaging the application as Fat Jar & Running the Fat Jar

Learn how to scale a Kafka Streams application

Scaling our Application

Wrap up on all the awesome learning from this section

Section Wrap-Up
+ KStreams and KTables Simple Operations (Stateless)
16 lectures 45:35

Learn the objective of this section, learning the stateless operations in Kafka Streams

Section Objectives

Learn about KStream and KTables 

KStream & KTables

Learn what Stateless and Stateful mean

Stateless vs Stateful Operations

Learn about the Map and MapValues Kafka Streams Operation

MapValues / Map

Learn about the Filter and FilterNot Kafka Streams Operation

Filter / FilterNot

Learn about the FlatMap and FlatMapValues Kafka Streams Operation

FlatMapValues / FlatMap

Learn about the Branch Kafka Streams Operation


Learn about the SelectKey Kafka Streams Operation


Learn how to read a KStream or KTable from Kafka

Reading from Kafka

Learn how to write your KStream or KTable back to Kafka

Writing to Kafka

Learn what operations mark streams for re-partition

Streams Marked for Re-Partition

Learn the fundamentals behind Log Compaction. This is a refresher!

Refresher on Log Compaction

Learn what is the KStream and KTable Duality

KStream and KTables Duality

Learn how to transform a KTable into a KStream in Kafka Streams

Transforming a KTable to a KStream

Learn how to transform a KStream into a KTable

Transforming a KStream to a KTable

Summary of all the learning on Stateless Operations

Section Summary
+ Practice Exercise - FavouriteColour
5 lectures 26:38

Learn about the practice Kafka Streams exercise, favourite colour!

FavouriteColour - Practice Exercise Description & Guidance

Hints and guidance to conduct the programming of this Kafka Streams application by yourself

Stuck? Here are some Hints!

Java solution for the Kafka Streams application FavouriteColour

Java Solution

Running the Favourite Colour Kafka Streams application

Running the application

Learn how to write the same application using Scala

Scala Solution
+ KStreams and KTables Advanced Operations (Stateful)
9 lectures 16:46

Section objective on KStream and KTable stateful operations

Section Objective

Learn about the groupBy operation

KTable groupBy

Learn the count operation

KGroupedStream / KGroupedTable Count

Learn the aggregate operation

KGroupedStream / KGroupedTable Aggregate

Learn the reduce operation

KGroupedStream / KGroupedTable Reduce

Learn the peek operation on KStream

KStream peek

Learn the Transform / Transform Values operation on KStream

KStream Transform / TransformValues

Learn how Kafka Connect can help you write your transformed data into a sink store / external system

What if I want to write to an external System?

Summary diagram from the Confluent documentation for operations

Summary Diagram
+ Exactly Once Semantics - Theory
4 lectures 12:21

Learn what Exactly Once Semantics are

What's Exactly Once?

Learn how Kafka 0.11 solves the problem to have exactly once semantics

Exactly Once in Kafka 0.11

Understand the use cases of At Least Once vs Exactly Once

What's the problem with at least once anyway?

Learn how to easily achieve exactly once semantics in Kafka Streams!

How to do exactly once in Kafka Streams
+ Exactly Once Semantics - Practice Exercise - BankBalance
7 lectures 34:56

Description for the BankBalance practice Kafka Streams exercise

BankBalance - Exercise Overview

Guidance for you to write your Kafka Producer in order to produce random bank transactions

Kafka Producer Guidance

Solution for the Kafka Producer that generates JSON random bank transactions

Kafka Producer Solution

Hints on how to write your Kafka Streams application

Kafka Streams Guidance & Hints

Solution to build the Bank Balance Kafka Streams application

Kafka Streams Solution

Let's run the Bank Balance producer and Kafka Streams application!

Running the BankBalance Application

Summary of our learnings to date

Section Summary
+ Joins - KStream to GlobalKTable example
6 lectures 26:11

Learn about Joins in Kafka Streams. Learn when it is or isn't possible to join data

What are joins in Kafka Streams?

Understand what is co-partition of data, join constraints, and how global KTables solve the problem

Join Constraints and GlobalKTables

Learn about the different types of joins in Kafka Streams

The different types of joins: Inner Join, Left Join, Outer Join

Hands On Example to perform a join between a KStream and a KTable

Creating a join with UserEnrich Kafka Streams App

Blog post from Confluent that explains joins semantics

All the joins explained - Blog Post

Running the example with an interactive producer that has different cases

Running the Kafka Streams Join application
+ Testing your Kafka Streams Application
5 lectures 22:54
Disclaimer: Advanced Lectures
Testing in Kafka Streams
Setup your Kafka Streams project
Hands-On: Test your WordCount application
Self-Guided Practice Exercises