KSQL for Stream Processing

Name: KSQL for Stream Processing
Rating: 4.6 (4 reviews)

KSQL Setup + Basics + Intermediate + Advance - step by step hands on!

Created byUp Degree

Last updated 12/2021

English

English [Auto],

What you'll learn

Learn all the fundamentals of KSQL
KSQL Environment Setup
Create Our first KSQL Stream
KSQL Datagen - Generating Streams
Deal with all kinds of data formats: CSV, JSON & Avro
Struct Types (Nested JSON)
Arrays and Maps
Create and use UDF, UDAF & UDTF

Course content

6 sections • 43 lectures • 2h 52m total length

Course curriculum Overview3:43
Explore KSQL for real-time stream processing, covering setup, streams and tables, joins, data encoding, partitioning and windowing, extensions, and production configurations, with Kafka fundamentals as your foundation.
Introduction to KSQL3:03
Explore ksql, the streaming sql engine for Kafka, offering interactive sql-like queries to filter, transform, and aggregate data across topics. It enables scalable, fault-tolerant processing with joins and window operations.
Use cases of KSQL3:05
Explore how ksql enables streaming edl, real-time analytics, and real-time monitoring in Kafka, with anomaly detection, personalization, and unified access across diverse data for IoT and customers.
Components of KSQL1:41
Explore the KSQL architecture with its three major components: the KSQL engine, the REST API, and the Kafka cluster connection, and how clients' statements and queries drive topology.
Download : Study Note

Installing Virtual Box2:47
install VirtualBox to run a linux environment on Windows or Mac, import a CentOS image, and set up essential software including a schema registry to work with secret.
Importing CentOS Image4:32
Import the CentOS image into VirtualBox, boot the VM, log in as root, and verify Kafka components installed via the Confluent open source distribution.
Installing MobaXterm6:26
Learn to install mobaxterm, set up a remote SSH session to a CentOS virtual machine, transfer files, manage multiple sessions, and monitor Confluent services for KSQL stream processing.
Download : study note

What is KSQL Datagen - Generating Streams3:41
Explore KSQL datagen to generate streaming test data for Kafka with schema-based options, presets, and formats such as JSON, Avro, and delimited, plus configurable bootstrap server, iterations, and intervals.
Working with CSV Delimited Data4:34
Handle csv delimited data with ksql for stream processing, pushing delimited messages to the impressions topic. Create a stream with fields like impression_time, user_id, and ad_id, then query and filter.
[code] working with CSV Delimited Data0:01
Working with JSON Data3:09
Explore handling json data with ksql: generate json events into the page_views topic, create a stream page_views_original with view_time, user_id, and page_id, and query the stream to verify the data.
[code] Working with JSON Data0:01
Working with Avro Data4:04
Learn to work with avro data in ksql by generating avro records, pushing them to a topic, and querying their avro value with a schema registry and console consumer.
[code] Working with Avro Data0:01
Working with struct type (nested json)3:00
Explore handling struct types and nested JSON in KSQL for stream processing, creating streams over fields, extracting city and state from address, and querying maps and arrays.
[code] Working with struct type (nested json)0:01
Working with Arrays and Maps4:07
Explore how to work with arrays and maps in KSQL for stream processing, including accessing map keys, array indices, and reading JSON messages from users_extended.
[code] Working with Arrays and Maps0:01
[study note] Working with Different types of Data

Manipulate a Stream12:47
Learn to create and describe a KSQL stream, query streams with select, apply numeric and string functions, and build a new stream from an existing one using persistent query.
[Download the Code] Manipulate a Stream0:02
Operations on Table9:07
Explore KSQL operations on tables, creating a materialized table from history, updating on key changes, and querying current state with group by, push and pull queries.
[code] Operations on Tables0:02
Build a rekeyed table8:40
Learn to build a rekeyed table by creating an original stream, a second stream with a new key, and a table on the new stream, enabling different key usage.
[code] Building a rekeyed table0:01
Updating KSQL Tables4:43
Explore how ksql tables update to store only the latest value per key, while streams retain all records, demonstrated with a warehouse location Kafka demo.
[code] Updating KSQL Tables0:01
Schema Evolution I Theory12:56
Explore how the schema registry enables schema evolution in Kafka, explain backward, forward, and full compatibility, manage versions and subjects, and use id values to serialize and deserialize binary data.
Schema Evolution II Hands On10:41
Explore Schema Registry operations in a hands-on session: register and view schemas, manage versions, and test backward and forward compatibility, then update and delete schemas.
[Download the code] Schema Evolution II Hands On0:02
[Study Note] for the Chapter

Working with UDF21:07
Design and deploy a custom udf in ksql by building a Java class with annotated methods and applying it to compute the maximum temperature from a delimited sequence.
[code] Custom UDF0:05
[study note ] UDF
Working with UDAF15:36
Learn to write, deploy, and test a user-defined aggregate function (UDAF) in KSQL, producing a comma-separated string of unique devices per location by grouping over a five-second tumbling window.
[code] UDAF0:05
[study note ] UDAF
Working with UDTF14:11
Explore how to build and test a custom udtf in ksql for stream processing, splitting temperature strings into multiple output records and applying a threshold such as 16.
[code] Custom UDTF0:05
[study note ] UDTF

Requirements

Fundamental understanding of Kafka
An open mind to learn

Description

For this course, We have partnered with KSQL expert Amit Ranjan to bring you the ultimate KSQL course.

In this course, We'll have taken a project-based approach. We will start from absolute basic then gradually we will cover the Intermediate + Advance - step by step hands-on!

KSQL Setup
Create Our first KSQL Stream
KSQL Datagen - Generating Streams
CSV Delimited Data
JSON & Avro Data
Struct Types (Nested JSON)
Arrays and Maps
Working with UDF
Working with UDAF
Working with UDTF

In this course, We'll have taken a project-based approach. We will start from absolute basic then gradually we will cover the Intermediate + Advance - step by step hands-on!

Why Join in the Course?

Step by step guide
Theory + Practical
A lot of Study Materials
Live coding Example
Code for Homework

This Course Also Comes With:

Lifetime Access to All Future Updates
A responsive instructor in the Q&A Section
Links to interesting articles, and lots of good code to base your next applications onto
Certificate of Completion Ready for Download

Choose the course that could improve your career by 5X!!!

Apache Kafka is one of the most popular programming languages today. With the release of Apache Kafka in a new version, this skill has become more in-demand than ever. If you are looking to boost your income, have fun challenges, and welcome change then this course is for you!

Who this course is for:

Architects who want to understand how KSQL fits into their solution architecture
Developers planning to learn and write their first KSQL applications
Anyone planning to learn KSQL

KSQL for Stream Processing

What you'll learn

Explore related topics

Course content

Introduction5 lectures • 12min

KSQL Setup4 lectures • 14min

Your first Programme1 lecture • 14min

Working with Different types of Data12 lectures • 23min

KSQL Basics (Streams & Table Operations)12 lectures • 59min

KSQL Extension9 lectures • 51min

Requirements

Description

Who this course is for: