Apache Kafka Series - KSQL on ksqlDB for Stream Processing !

Download and unzip the course project files and PDF, ensuring you use the latest notes that align with the Confluent platform. Review the commands before copying to follow along.

Install Confluent platform to set up ksqlDB, configure PATH and CONFLUENT_HOME, then run the server locally (localhost:8088) on mac, linux, or windows (WSL2 or docker).

Create a ksqldb stream from a Kafka topic using comma-delimited data (name and country code), and explore push queries, emit changes, offset reset earliest, and simple aggregates.

Create a user profile stream using json in ksql for a taxi booking app, learn json field specification, describe streams, and query json data with integer, character, and double types.

Generate synthetic user profile data with Ksql datagen to feed streams, producing randomized records (user id, first name, last name, country, rating) in json or avro formats.

Extend streams with KSQL by building a stream on top of another, using a case statement to bucket ratings, and running scripts to define, drop, and monitor streams.

Explore ksql joins by joining a user profile stream with a country table using a left join on country code; create a new stream that enriches profiles with country names.

Learn how pull queries in KSQL DB retrieve immediate results from an aggregate table by querying the row key, contrasting with push queries that emit changes.

Explore how to control Kafka using ksqldb, configuring Kafka Connect sources and sinks with the ksql CLI, using a Postgres example to stream data into and out of Kafka.

Explore csv, json, and avro data encodings for taxi complaints, examining schema, compression, and trade-offs while modeling a four-field stream: customer name, complaint type, trip cost, new customer flag.

Create a comma-delimited complaint stream on the complaints csv topic in ksqlDB with fields customer name, complaint type, trip cost, and new customer flag; validate data and monitor server logs.

Transition from comma-delimited data to json to handle embedded commas and build a json-based complaints stream and topic for cleaner pipelines.

Explore Avro schema evolution by adding a new field, number of rides, to the complaints Avro topic, and compare version histories with side-by-side streams in Confluent Control Center.

Process a stream of weather events with nested json fields in ksqlDB, using a city struct to access name, country, latitude, and longitude via struct syntax.

Learn to fix join failures by repartitioning streams to achieve co-partitioning, create a rekeyed stream with a controlled partition count, and successfully join driver profile with the country table.

Explore geospatial functions in ksql, computing distances between rider and destination using geo distance, and join streams with weather data to report travel distance and conditions.

Extend ksqlDB with user defined functions and aggregates by building UDF/UDAF jars in Java with Maven, and register them in the extension directory.

Apply a Java-based user defined function taxi in ksqlDB to calculate taxi wait times from weather and distance, generating a customer-focused report of wait times, distance, and forecast.

Contrast interactive and headless deployments for ksqlDB, build a single application file, deploy in headless mode, and verify results via topics and the Kafka Avro Console Consumer.

Explore explain plans in ksqlDB to understand query execution from the show queries output, and visualize the topology to see data flow through streams to topics.

Explore how Kafka Streams enables scaling out and fault-tolerant processing across multiple KSQL instances, sharing broker lists and a common KSQL Server ID, with Docker-based high-availability testing.

Configure KSQL server and session settings via the properties file or set command, then verify with list properties noting server versus session scope and the service ID.

Explore how Kafka streams use RocksDB-backed state stores for stateful operations like windows, joins, and aggregates, and how the state dir affects disk storage and fault tolerance.

Test ksqldb applications with the Ksql Test Runner, using input and output scripts to drive test cases and enable test driven development without requiring a ksqldb or Kafka cluster.

Explore Ksql recipes and documentation, learn by examples like merging strings with insert into and Json format, consider migrating Kafka Streams to Ksql, and celebrate completing the course.

Encourage learners to leave reviews and feedback and rate the course on Udemy, and explore the bonus lecture listing coupons for other courses, promoting affordable education.