Realtime Analytics with Apache Pinot and Apache Flink

Explore real-time analytics for an online cab service, tracking rides in the past hour, cancellations over a year, city breakdowns, and revenue per user to drive rapid business decisions.

Learn to produce and consume data with Apache Kafka using the console producer and consumer, handling keys and values with a separator and the print-key option.

Explore Flink stream processing concepts such as ingestion time, event time, and processing time. Connect to Kafka with source and sink connectors and use Flink SQL for processing.

Read data from Kafka with Flink table API, create a stream table environment, execute a query on a Kafka-backed table, convert to a data stream, and print.

Explore how regular joins work in Flink, including inner, left outer, right outer, and full outer joins, with interval joins over time windows, treating streams as tables.

Learn to run Flink on a cluster in standalone mode, install locally, and deploy a project with dependencies or as code only, then validate with Kafka and MySQL inputs.

Pinot is a real-time distributed OLAP data store that ingests streaming data with low latency, supports batch ingestion, and enables user-facing analytics and dashboards with tools like Superset and Tableau.

Set up Apache Pinot on your local system by installing OpenJDK 11, downloading the Pinot binary, extracting it, and starting zookeeper, controller, broker, and server to run Pinot on localhost:9000.

In this hands-on lecture, ingest streams into Apache Pinot by creating a Kafka topic, producing messages, and loading them into a real-time table via the Pinot user interface or api.

Ingest enriched ride data from Flink into Pinot using a real-time upsert table with geospatial indexing, via Swagger REST APIs, Kafka, and H3-based coordinates.

Learn to run queries against Pinot using the REST API, craft SQL queries with curl, and visualize JSON results with jq, including handling single quotes and interpreting the response.

Set up superset locally by installing dependencies, creating a Python virtual environment, and installing the Pinot connector, then run the UI on localhost:8088 to manage dashboards and datasets.

This bonus lecture provides a deeper look at Pinot and promotes an in-depth Apache Pinot hands-on course, plus a real-time streaming course featuring Spark Streaming, Kafka Streams, and Druid.