Confluent ksqlDB complete course-Build streaming app by SQL

Name: Confluent ksqlDB complete course-Build streaming app by SQL
Rating: 4.6 (17 reviews)

Kafka streams,Kafka Connect,The database purpose-built for stream processing applications. Build streaming app by sql

Created byAlexander Wong

Last updated 5/2023

English

English [Auto],

What you'll learn

Confluent ksql&ksqlDB
Kafka streaming process application
Advanced Stream Processing with ksqlDB
ksqlDB and Apache Kafka connect component integration
One mental model for the entire stack
Build a complete streaming app with a few SQL statements
Confluent ksqlDB components and architecture
Develop UDFs and UDAFs
ksqldb Clients(REST API & Java Client)

Course content

8 sections • 68 lectures • 5h 49m total length

Course overview Introduction and agenda2:50
Explore ksqlDB fundamentals, set up the local environment, and build a complete streaming app using SQL statements, covering basic stream processing, collections, window statements, joins, aggregations, and Kafka Connect integration.
Prerequisites0:44

Meet ksqldb section content overview1:28
What is ksqlDB2:08
You can use ksqlDB to build event streaming applications from Apache Kafka® topics by using only SQL statements and queries. ksqlDB is built on Kafka Streams, so a ksqlDB application communicates with a Kafka cluster like any other Kafka Streams application.
ksqlDB Features and Benefits3:04
ksqlDB vs kafka streams framework2:41
ksqlDB Architecure and internal components5:08
Explore ksqlDB architecture and internal components, including the stream processing engine, materialized views, and the REST API for building streaming apps by defining statements with connectors and Kafka topics.
ksqlDB cluster deploy mode(interactive and headless)7:16

ksqlDB installation & quick start contents overview1:22
Apache Zookeeper & Kafka Cluster installation7:09
Install and configure zookeeper and kafka cluster for a single-instance setup, start zookeeper and kafka, then create a test topic and perform producer-consumer testing.
Confluent ksqlDB server installation5:48
Create the first streaming application by ksql statement6:41
Define variables and configurations, create a source stream, build the topology, and start the Kafka streams using ksql statements, with hands-on ddl and query demos.
Run sql scripts file at ksqlDB(batch submit queries statement)4:15
Execute Specific Statements and Quit3:49
ksqlDB headless deploy mode(lock down the Interactive)7:12
Demonstrate ksqlDB headless deploy mode, including setting properties, creating streams, deriving the user stream, and verifying results by consuming the transformed user topic.
ksqlDB query statement execution lifecycle2:02
Understand the ksqlDB query statement execution lifecycle from creating streams or tables with sql to building logic and physical plans, then running and managing the streaming application.
Which SQL operations are supported in headless mode
Run a list of predefined queries and a file by using RUN SCRIPT

Stream processing basics with ksqlDB contents overview1:56
ksqlDB simple data types and compound data types9:51
Learn ksqlDB simple and compound data types, modeling a person stream with name, age, and address as a struct and friends as a map or array.
ksqlDB support custom brand-new data types2:39
Learn how ksqlDB supports custom data types using the create tabs statement to define address as a structure and reuse type definitions while managing streams.
ksqlDB Serialization and Serialization Formats1:45
ksqlDB DELIMITED Serialization Formats process CSV Data4:39
ksqlDB JSON_SR Serialization Formats and Confluent Schema Registry Installation4:57
Compare ksqlDB json and adjacent sr formats, and learn to install and configure confluent schema registry, connect it to Kafka, set the _schemas topic, and verify integration.
Lab: ksqlDB integration with Schema Registry process JSON_SR Formats Data5:43
ksqlDB collections introduction: Stream & Table4:34
Explore ksqlDB collections by comparing streams and tables, where streams are immutable event sequences and tables store the latest state with tombstones, lookup tables, and derived collections.
ksqlDB create source collections(streams&tables) syntax and examples-I7:03
ksqlDB create source collections(streams&tables) syntax and examples-II4:28
ksqlDB time semantics and custom timestamp column2:16
Lab: ksqlDB time semantics and custom timestamp column6:25
Manipulation Collections(Show/Describe/Alter/Drop)5:28
ksqlDB collections basic queries and examples7:27
Conditional Expression in collection where clause queries(handle NULL value)6:01
Creating Derived Collections(CSAS and CTAS) syntax and examples6:46
Persistent Queries-Writing Results Back to Kafka target topic3:53
Lab: ksqlDB implement WordCount application by ksql query statement7:42
Explain statement - Show the execution plan for a SQL expression2:20
Explore how the explain statement reveals the execution plan and streams topology for a sql expression, showing source text line streams, flat map transforms, and sink to the word topic.
Lab: ksqlDB implement XMall transaction data real-time analysis application9:07
Extra: Defines a variable to be used within SQL statements7:26
Define variables and substitution in ksqlDB statements, including proper single-quote quoting and value substitution. Use these variables to create streams with specified columns, partitions, and formats.
About Streams and Tables, which statement is incorrect?

Advanced Stream Processing with ksqlDB and Kafka Connect contents overview1:42
ksqlDB collections joins and windowed3:05
Explore ksqlDB collection joins and windowed processing for streams and tables. Learn inner, left, and full joins and how windowing applies to unbounded streams.
ksqlDB stream join(inner/left/full) another stream with windowed7:34
Demonstrates join operations in ksqlDB by joining two streams with a one-minute window: inner, left, and full joins, using stream schemas, timestamps, and on conditions to produce joined results.
ksqlDB stream join(inner/left) another table5:11
ksqlDB table primary key join(inner/left/full) another table6:02
Demonstrates ksqlDB table primary key joins, including inner, left, and full joins, and contrasts them with foreign key joins, while noting windowed joins and eventual consistency.
ksqlDB table foreign key join(inner/left) another table4:26
ksqlDB collections grouping clause & aggregating function2:44
The group by clause basic queries and examples6:06
Explore the group by clause with having, counting people by title and extracting the top per title, while noting aggregate limitations and stream versus table behavior in Kafka streams.
Which sql statement is correct in group by clause?
Lab: ksqlDB grouping clause build sales champion application9:53
ksqlDB time and windows2:17
Which type of time window not support in ksql&ksqlDB
ksqlDB Tumbling time window and network potential attack analysis by realtime7:15
ksqlDB suppress windowed intermediate computation result to downstream(suppress)3:02
ksqlDB Hopping time window and analysis Twitter hot words by realtime5:34
ksqlDB Session time window and analysis web visit stats during a valid session2:44
Apache Kafka Connect introduction4:50
Apache Kafka Connect distributed cluster setup5:20
Configure and start a Kafka Connect distributed cluster, set bootstrap servers and group ID, enable plugins, verify with REST health checks, and prepare to register connectors for ingesting filesystem data.
Apache Kafka connect install connector plugins and create FileSource Connector7:37
ksqlDB integrate Kafka Connect distributed cluster define FileSource connector6:33
Heartbeat sensor data realtime analysis patient health monitor-JDBC Connector7:58
Install and configure the JDBC source connector for Kafka Connect, create two connectors to ingest patient and room table data into Kafka topics for real-time health monitoring and alerting.
Lab: Heartbeat sensor data real-time analysis patient health monitoring10:26
ksqlDB Materialized Views4:19

ksqlDB Functions(UDF & UDTF &UDAF) contents overview2:17
Learn how to develop ksqlDB user defined functions, including scalar UDFs, UDTFs, and UDAFs, and apply lambda functions in SQL.
ksqlDB function custom Scalar functions(UDF) implement compute BMI formula10:05
Learn to implement a bmi udf as a scalar function in ksqldb by creating a java maven project, annotating the function, and deploying it as an extension for the server.
ksqlDB function custom Tabular functions(UDTF) implement split_and_explode6:37
Define a custom ksqlDB udtf (tabular function) that outputs multiple rows from a single input by using split and explode on a delimiter.
ksqlDB function custom Aggregation functions(UDAF) implement MAX2 applied table7:51
ksqlDB Use lambda functions7:33

ksqlDB REST API5:33
Interact with the local ksqlDB using the rest api to list streams and tables, create or drop resources, run push and pull queries, insert data, and monitor health and properties.
ksqlDB Java Clients - interactive ksqlDB collections by Java code2:09
Explore a lightweight Java client for ksqlDB that sends requests from your Java app, supports pull and push queries, and manages streams, tables, and persistent queries.
ksqlDB Java Client Example(list streams & asynchronous query)10:15
Demonstrates using the ksqlDB java client to list streams and print stream and topic details, then executes a push query with a subscriber for asynchronous results.
ksqlDB Java Client Example(synchronous query & batch query)6:47
Learn how to use the ksqlDB Java client to execute synchronous stream queries and batch pull queries, print results to the console, and manage the client lifecycle.

Requirements

Apache Kafka fundamental
Apache Kafka Streams framework fundamental
SQL fundamental
JVM(Java/Scala) development experiences

Description

ksqlDB is a database purpose-built for stream processing applications. What exactly does this mean? It incorporates many components found in almost every stream processing architecture.

This is important because almost all streaming architectures today are piecemeal solutions cobbled together from different projects. At a minimum, you need one subsystem to get events from existing data sources, another to store them, another to process them, and another to query against aggregation. Integrating each subsystem can be difficult. Everyone has their own mental models. Given all these complexities, it's easy to wonder: is it all worth it?

ksqlDB aims to provide a mental model to do everything you need. You can build a complete streaming application against ksqlDB, which has only one dependency: Apache Kafka®.

This course will comprehensively and meticulously introduce all the knowledge points of ksql and ksqlDB, allowing you to master the use of ksqlDB from beginner to proficient, from shallow to deep. Using ksqlDB, you can easily use a few lines of SQL statements to complete the development of streaming computing applications for real-time data.

[course requirements]

Some Apache Kafka/Confluent Kafka experience is required
Procedural experience with relational database SQL
Familiar with the Kafka Streams framework (kstream, ktable)

[course features]
Practise driven
Lots of real cases trainning
Step by step shallow to deep
The course content are straightforward
Covers most Confluent ksqlDB & ksql content
Rich comprehensive case help you fully understand

Who this course is for:

Build a complete streaming app with a few SQL statements
Build Complex Event Processing(CEP) application by ksqlDB
Master Confluent ksql&ksqlDB
Master Realtime streaming event process by sql

Confluent ksqlDB complete course-Build streaming app by SQL

What you'll learn

Explore related topics

Course content

Course Introduction2 lectures • 4min

Meet Confluent KSQL & ksqlDB6 lectures • 22min

ksqlDB installation & quick start8 lectures • 38min

Stream processing basics with ksqlDB21 lectures • 1hr 52min

Advanced Stream Processing with ksqlDB and Kafka Connect21 lectures • 1hr 55min

ksqlDB Functions(UDF & UDTF &UDAF)5 lectures • 34min

ksqlDB Clients4 lectures • 25min

PowerPoint and Source Code Download1 lecture • 1min

Requirements

Description

Who this course is for: