Confluent ksqlDB complete course-Build streaming app by SQL

Explore theoretical knowledge of covalent, physical and keizoku, and the features of key circle and key tokudb. Examine ksqldb and the Kafka stream framework, internal components, deployment models—interactive and headless.

Explore ksqlDB features and benefits, including building event streams with statements, using materialized views for lookups, and simplified architectures that reduce code and enable interactive data exploration.

Explore how ksqlDB and the Kafka Streams framework cooperate to build streaming apps on Kafka topics, comparing SQL-driven processing with low-level stream processing for flexible stateful operations.

Explore ksqlDB architecture and internal components, including the stream processing engine, materialized views, and the REST API for building streaming apps by defining statements with connectors and Kafka topics.

Learn how to deploy a ksqlDB cluster in interactive or headless modes, connect via REST and CLI, and manage persistent queries with command topics and configuration topics.

Learn to set up the ksqlDB development environment with Zookeeper, Kafka, and ksqlDB server clusters, run your first streaming app, and grasp the ksqlDB query lifecycle.

Install and configure zookeeper and kafka cluster for a single-instance setup, start zookeeper and kafka, then create a test topic and perform producer-consumer testing.

Install and configure the Confluent ksqlDB server in a local environment, set service and listener properties, start the server, verify with a health check, and prep for the streaming app.

Define variables and configurations, create a source stream, build the topology, and start the Kafka streams using ksql statements, with hands-on ddl and query demos.

Learn to batch submit queries in ksqlDB using the run script command, create streams and tables, and run predefined queries from a script file.

Learn how to execute specific statements in ksqlDB via file options and inline commands, then create streams, query the streams, and quit the session.

Demonstrate ksqlDB headless deploy mode, including setting properties, creating streams, deriving the user stream, and verifying results by consuming the transformed user topic.

Understand the ksqlDB query statement execution lifecycle from creating streams or tables with sql to building logic and physical plans, then running and managing the streaming application.

Explore stream processing basics with ksqlDB, from data types and schema registry to source collections and derived queries, and build a word count app using persistent queries and Kafka topics.

Learn ksqlDB simple and compound data types, modeling a person stream with name, age, and address as a struct and friends as a map or array.

Learn how ksqlDB supports custom data types using the create tabs statement to define address as a structure and reuse type definitions while managing streams.

Explore ksqlDB serialization concepts, learn how to choose key or value formats when creating streams or tables, and compare JSON values with schema registry integration, CSV, and protocol buffers.

Explore ksqlDB's delimited serialization format for csv data by defining key and value delimiters, including the default comma, creating a topic and stream, and querying records from a csv-based stream.

Compare ksqlDB json and adjacent sr formats, and learn to install and configure confluent schema registry, connect it to Kafka, set the _schemas topic, and verify integration.

Integrate confluent schema registry with ksqlDB and use the json schema registry serialization format to produce employee records via a Java producer and create a stream with json schema.

Explore ksqlDB collections by comparing streams and tables, where streams are immutable event sequences and tables store the latest state with tombstones, lookup tables, and derived collections.

Learn ksqlDB source streams and tables syntax through practical examples, including create, replace, not exists handling, and querying fields and metadata. Discuss read-only source behavior and a pull query discrepancy.

Create ksqlDB source collections for streams and tables, using parameter keys and primary keys, note permanent keyword differences, and build tables via select all or post queries with multiple keys.

Define ksqlDB streams and tables with clause to set a custom timestamp column and its format, and understand event time, injected time, and process time for windowing and joins.

Learn how to control ksqlDB time semantics by using a custom timestamp column, specifying time step and timestamp format, and deriving streams to override defaultKafka record timestamps.

Explore how to view, describe, alter, and drop streams and tables in Confluent ksqlDB, with show and list commands and cascade options for topics.

Explore ksqlDB basics by filtering and transforming data in the person stream with transient queries, selects, where conditions, between, limit, and export to individual records.

Master coalesce, if, not, and case expressions to handle nulls in collection where clause queries, then apply these techniques to build streaming apps with SQL.

Create derived collections by deriving streams and tables from existing ones using CSAS and CTAS syntax, including source collections, partition by, and persistent queries that recover after restarts.

Explore persistent queries in ksqlDB by writing results back to a Kafka topic, explicitly naming topics, configuring partitions and formats, and creating streams and tables based on derived collections.

Learn to implement a word count app with ksqlDB by creating a source stream from text line, flat-mapping to words, grouping, counting, and materializing results to a topic.

Explore how the explain statement reveals the execution plan and streams topology for a sql expression, showing source text line streams, flat map transforms, and sink to the word topic.

Build a ksqlDB based XMall transaction real-time analysis app by creating source and derived streams, masking data, extracting fields, calculating rewards, and routing to the XML electronic topic.

Define variables and substitution in ksqlDB statements, including proper single-quote quoting and value substitution. Use these variables to create streams with specified columns, partitions, and formats.

Advance stream processing with ksqlDB and Kafka Connect, covering joins, grouping, having, aggregates, and window types, plus setting up the Kafka Connect Worker cluster to build a complex streaming app.

Explore ksqlDB collection joins and windowed processing for streams and tables. Learn inner, left, and full joins and how windowing applies to unbounded streams.

Demonstrates join operations in ksqlDB by joining two streams with a one-minute window: inner, left, and full joins, using stream schemas, timestamps, and on conditions to produce joined results.

Inner and left joins between a stream and a table are supported by ksqlDB, using lookup semantics; outer joins aren’t available, and downstream outputs react only to stream updates.

Demonstrates ksqlDB table primary key joins, including inner, left, and full joins, and contrasts them with foreign key joins, while noting windowed joins and eventual consistency.

Demonstrate foreign key joins in ksqlDB by linking two tables with a foreign key extractor, covering inner and left joins and noting full outer joins aren’t supported for this setup.

Explore how ksqlDB uses the group by clause and aggregate functions to compute summaries over streams and tables, storing results in mutable structures for live updates.

Explore the group by clause with having, counting people by title and extracting the top per title, while noting aggregate limitations and stream versus table behavior in Kafka streams.

Apply ksqlDB grouping clause to build a sales champion app by aggregating sales amounts per department, joining salesman data, and deriving streams and persistent queries for top sales.

Learn how ksqlDB handles time and windows to support stateful aggregations and joins. Define windows by start and end, and explore hopping, tumbling, and central windows for grouping by key.

Implement ksqlDB tumbling time windows for real-time network attack analysis, counting visits per remote address in one-minute intervals to detect potential threats using a ten-visit threshold.

Explore how ksqlDB and Kafka Streams handle windowed aggregations, emitting final results only when a window closes, using the suppress operation to withhold updates to downstream systems.

Configure hopping time windows in ksqlDB by setting a one-minute window size and a 30-second hop interval to analyze real-time hot words, such as Twitter keywords, with overlapping windows.

Explore ksqlDB session time window dynamics where activity shapes windows and inactivity gaps merge or start new sessions, using a 60-second inactive gap and practical examples.

Introduce Kafka Connect and its integration with Ksqldb, detailing source and sink connectors, tasks, workers, converters, and external vs embedded deployment models for scalable, fault-tolerant data streams.

Configure and start a Kafka Connect distributed cluster, set bootstrap servers and group ID, enable plugins, verify with REST health checks, and prepare to register connectors for ingesting filesystem data.

Install and configure file source plugins for Kafka Connect, restart the worker, and create a FileSource connector to stream files into a topic with schema metadata visible.

Integrate the Kafka Connect distributed cluster with SQL Server by defining a file source connector, configure properties, restart the DB, and verify data ingested into a Kafka topic.

Install and configure the JDBC source connector for Kafka Connect, create two connectors to ingest patient and room table data into Kafka topics for real-time health monitoring and alerting.

Build a real-time heartbeat monitoring workflow with ksqlDB by integrating JDBC sources from MySQL into Kafka, generating a windowed heartbeat stream and threshold-based alerts to doctors.

ksqlDB materialized views pre-compute and store query results to enable fast reads, updating in real time via CDC and incremental data.

Learn how to develop ksqlDB user defined functions, including scalar UDFs, UDTFs, and UDAFs, and apply lambda functions in SQL.

Learn to implement a bmi udf as a scalar function in ksqldb by creating a java maven project, annotating the function, and deploying it as an extension for the server.

Define a custom ksqlDB udtf (tabular function) that outputs multiple rows from a single input by using split and explode on a delimiter.

Discover how to build a ksqlDB custom aggregation function (udaf) that maintains a stateful maximum and can be applied to a table, with initialize, aggregate, merge, map, and undo.

Explore ksqlDB lambda functions to transform, filter, and reduce collections. Apply inline functions to streams to uppercase course names, increment scores, and reduce arrays to a single value.

Interact with the local ksqlDB using the rest api to list streams and tables, create or drop resources, run push and pull queries, insert data, and monitor health and properties.

Explore a lightweight Java client for ksqlDB that sends requests from your Java app, supports pull and push queries, and manages streams, tables, and persistent queries.

Demonstrates using the ksqlDB java client to list streams and print stream and topic details, then executes a push query with a subscriber for asynchronous results.

Learn how to use the ksqlDB Java client to execute synchronous stream queries and batch pull queries, print results to the console, and manage the client lifecycle.