Realtime Analytics with Apache Pinot and Apache Flink

Explore real-time analytics for an online cab service, tracking rides in the past hour, cancellations over a year, city breakdowns, and revenue per user to drive rapid business decisions.

Explore the data model and architecture for real-time analytics, detailing rides, riders, drivers, and location datasets, their joins, and how Flink enriches data for Pinot and Superset visualizations.

Learn a quick introduction to Apache Kafka as a distributed, horizontally scalable event streaming system with topics, partitions, replication for high availability, and multiple producers and consumers.

learn to set up kafka locally, install java, download kafka 3.6.1, run without zookeeper, configure advertise listeners and log dirs, and create topics rights, riders, drivers, and rights enriched.

Learn to produce and consume data with Apache Kafka using the console producer and consumer, handling keys and values with a separator and the print-key option.

Explore Flink stream processing concepts such as ingestion time, event time, and processing time. Connect to Kafka with source and sink connectors and use Flink SQL for processing.

Read data from Kafka with Flink table API, create a stream table environment, execute a query on a Kafka-backed table, convert to a data stream, and print.

apply basic transformations in flink sql by selecting id, amount, and lowercasing status, then run a flink sql example with kafka and compare to the stable api workflow.

Learn how Flink uses checkpoints to resume processing exactly where it left off after a failure. Understand how watermarks manage late data and latency in streaming analytics.

Explore how regular joins work in Flink, including inner, left outer, right outer, and full outer joins, with interval joins over time windows, treating streams as tables.

Learn how Flink's temporal joins use versioned tables to join streams as of a specific event or processing time, including lookup joins with external data sources.

Install and start MySQL, create the analytics database and an analytics user, then create a location table with location_id as the primary key and insert dummy data for Flink joins.

enrich ride data in real time by performing event-time temporal joins with drivers and riders and a lookup join with location, using Flink and Kafka.

Learn to run Flink on a cluster in standalone mode, install locally, and deploy a project with dependencies or as code only, then validate with Kafka and MySQL inputs.

Pinot is a real-time distributed OLAP data store that ingests streaming data with low latency, supports batch ingestion, and enables user-facing analytics and dashboards with tools like Superset and Tableau.

Set up Apache Pinot on your local system by installing OpenJDK 11, downloading the Pinot binary, extracting it, and starting zookeeper, controller, broker, and server to run Pinot on localhost:9000.

In this hands-on lecture, ingest streams into Apache Pinot by creating a Kafka topic, producing messages, and loading them into a real-time table via the Pinot user interface or api.

Explore how to perform upserts in Pinot, using a primary key, choosing full or partial upsert modes, and optional delete operations, with snapshot and preload considerations.

Ingest enriched ride data from Flink into Pinot using a real-time upsert table with geospatial indexing, via Swagger REST APIs, Kafka, and H3-based coordinates.

Execute real-time analytics queries in Pinot using Pinot Data Explorer to analyze rides over the last hour, including counts, sums, averages, group by, and geospatial radius calculations.

Learn to run queries against Pinot using the REST API, craft SQL queries with curl, and visualize JSON results with jq, including handling single quotes and interpreting the response.

Learn Apache superset as an exploration and visualization tool that connects to databases like Pinot and MySQL, enables SQL queries, and creates charts and dashboards from real-time data.

Set up superset locally by installing dependencies, creating a Python virtual environment, and installing the Pinot connector, then run the UI on localhost:8088 to manage dashboards and datasets.

Connect Superset to Apache Pinot by configuring a SQLAlchemy URI, testing the connection, creating a writes enriched data set, and validating real-time queries from Pinot via Superset.

Create a real-time analytics dashboard in Superset by building and combining charts from live data in Kafka, processed by Flink and stored in Pinot, then visualize it in a dashboard.

Learn to perform real time analytics with Flink and Pinot, including joins, connecting Pinot to Kafka, running real time queries, and building dashboards with Superset.

This bonus lecture provides a deeper look at Pinot and promotes an in-depth Apache Pinot hands-on course, plus a real-time streaming course featuring Spark Streaming, Kafka Streams, and Druid.