Apache Flink Relational Programming using Table API and SQL

Explore how Apache Flink powers batch and streaming data pipelines through the PyFlink Python API, using the table API and SQL interface to return table objects and integrate with pandas.

Explore how Apache Flink's table API and SQL interface enable relational data processing with high-level languages like Python, Java, and Scala, via select, group by, and filter.

Discover the table data structure in Apache Flink as a lazy, relational abstraction used by the table API and SQL to query sources, chain operations, and produce results.

Explore Flink's metadata repository and the default ephemeral catalog that stores databases, tables, partitions, views, and UDFs, and learn to use external catalogs, Hive metastore, JDBC access, and custom catalogs.

Explore how Apache Flink unifies batch and stream processing; batch uses bounded data for global stats, while stream handles unbounded data with windowed time aggregations.

Install Apache Flink by choosing Java 11 and Python 3.8. Install a JDK, not just a JRE, and use PyFlink via pip in a virtual environment.

Install PyFlink by setting up a Java 11 environment, creating a Python virtual environment, and installing Apache Flink bindings via a requirements file to meet version 1.13–1.14.

Understand the table environment as the entry point to Flink for batch and streaming. Use it to register catalogs, tables, and UDFs, and to execute SQL with the Blink planner.

Learn to create table environments in Flink using environment settings to build batch and streaming table environments, and integrate the table API with the Datastream API for seamless relational programming.

Explore source tables in Apache Flink with the Table API and SQL to read data and write results; create sources from hardcoded iterables or files, databases, and queues like Kafka.

Create a source table in Apache Flink from a Python iterable using from_elements, exploring the table API, schema definitions, and pandas conversion.

Create csv source tables in Apache Flink by defining a schema and registering a source table in a table environment, then read and print the schema and data from file.

Demonstrates creating an Apache Flink Kafka source table with the table API to read from a Kafka topic, then stream data to a black hole sink.

Explore sink tables in Apache Flink relational programming, enabling you to write sourced and processed data to external systems such as files, database tables, and message systems.

Demonstrates reading data from a CSV source and writing to a CSV sink using Apache Flink's table API, including defining source and sink tables and running a batch job.

Demonstrates creating a Kafka sink table in Apache Flink with the Table API and SQL, streaming data from a source topic to a sink topic.

Compare static bounded batch tables and unbounded streaming tables in Apache Flink’s relational programming with table API and SQL, highlighting continuous processing and unified operations across modes.

Explore how data flows into and out of Apache Flink using Table API and SQL for both bounded batch and unbounded streaming, highlighting windowing, row time, and state stores.

Explore the basics of table API and SQL operations in Flink: projection, selection, filtering, and grouped aggregations via group by, in a fluent, non-mutating, chainable style.

Explore selecting columns and deriving sales in Apache Flink with the table API and SQL, including distinct selections and computing daily revenue from price and quantity.

Filter data with where and filter on a derived sales value (≥80) using the Apache Flink table api and interface, and compare with an inner query for date-based predicates.

Explore joining tables in Apache Flink using the Table API and SQL, performing inner and outer joins on sales and seller locales, with practical, example-driven guidance.

Explore aggregations in Apache Flink with the Table API and SQL, computing average prices by product and revenue from price times quantity via group by seller.

Explore how Apache Flink handles real-time streaming with event time and processing time, comparing their roles in unbounded data, and learn about watermarking for late or out-of-order events.

Explore streaming aggregations and windowing to perform moving-window statistics on unbounded data streams over a time dimension, enabling up-to-date analytics and event-driven architectures.

Explore tumbling windows in stream processing by defining non-overlapping time blocks, grouping events within each 60-second interval, and computing the block-specific results.

Explore sliding windows to compute revenue over the last 60 seconds with 20-second updates, offering frequent real-time insights, and contrast them with tumbling windows.

Explore Apache Flink session windows, a windowing type bounded by inactivity gaps, illustrated with order streams and 30-second gaps. Learn to implement these windows with Table API and SQL.

Set up a real-time data pipeline with Confluent Kafka using Docker, run a Python sales producer to populate sales items topic, and explore Flink windowing with Table API and SQL.

Explore tumbling window aggregations with the Apache Flink table API, using the COFCO connector to process sales data in 30-second windows and compute revenue per seller.

Explore tumbling windows in Apache Flink using SQL and the table API, consuming from Kafka via the COFCO connector and writing windowed revenue to a Kafka sink every 30 seconds.

Demonstrates sliding window aggregations in Apache Flink using the table API, calculating 30-second revenue with 10-second updates from a Kafka stream and outputting results back to Kafka.

Demonstrates sliding (hopping) window aggregations in Apache Flink using SQL and the Table API for relational programming, processing sales data from Kafka and outputting windowed results back to Kafka.

Demonstrates session windowed aggregations in Apache Flink using the table API, defining 20-second inactivity windows with processing time, and outputting results to Kafka topics.

Demonstrates session windows in Apache Flink with SQL, showing inactivity defining windows via processing time and streaming data from Kafka through the table API.

Explain event time processing in Apache Flink, using watermarks to handle late or out-of-order events, and demonstrate event time tumbling windows and source definition with to_timestamp.

Explore row operations and user defined functions in the Apache Flink table API with Python to compute total annual sales, the quarterly average, and the standard deviation.