Apache Druid : Complete Guide

Druid is a time series database designed for real-time analytics and fast slice-and-dice queries on large datasets, using columnar storage, time-based partitioning, and a scalable distributed architecture for real-time dashboards.

Install and run a standalone Apache Druid deployment locally, access the Druid portal, and explore broker, coordinator, historical, middle manager, and router for real-time queries with ingestion options.

Configure Apache Druid to use an external MySQL metadata storage. Install MySQL, create the druid user and database, enable the MySQL metadata storage extension, and restart Druid.

Configure S3 as Apache Druid's deep storage to back up data chunks, update runtime properties, enable the S3 extension, provide access keys, and restart the service to enable deep storage.

Install and configure Kafka and the Twitter producer app, stream real-time tweets into Apache Kafka, and ingest and flatten the data into Apache Druid for live analytics.

Ingest twitter data into Apache Druid via Apache Kafka, configure bootstrap server and topic, stream from the beginning, and parse JSON to flatten nested columns.

Explore Apache Druid transformation and filter functions, including upper case conversion and various filters—selector, in, interval, regex, like, search, and JavaScript—for shaping and querying data.

Configure Apache Druid for real-time Kafka ingestion, map column types, manage rollup and segment granularity, and optimize query performance through tuning and pre-aggregation.

Explore architecture of Apache Druid, detailing how middle managers ingest data, create mutable segments, and publish to deep storage, with coordinators, overloads, historical nodes, and brokers managing availability and queries.

Explore how Apache Druid stores segments in S3 deep storage, detailing the bucket structure, data folder, twitter_data table, and hourly segments with partitioned, indexed, and zip-compressed segment files.

Explore Apache Druid tuning across partition, tune, and publish stages, focusing on segment granularity, maximum rows per segment, and total rows to optimize real-time dashboards.

Explore continued Apache Druid tuning by configuring memory limits, disabled rollup, offset handling, timeouts, persistence, handoffs, worker threads, index bitmap type rowring, and compression options.

Query Apache Druid from an external application using the Avatica JDBC driver by configuring a JDBC connection and adding the Maven dependency to run a Twitter data query.

Demonstrate schema evolution in druid by ingesting json data via kafka, adding new columns, configuring timestamp, and validating data through a supervisor task and ingestion pipeline.

Explore evolving the druid schema by adding manager_id and department_id, configuring schema, re-ingesting via kafka, and applying updates with minimal downtime.

Explore Apache Druid batch ingestion, including native, SQL, and Hadoop-based options, and learn to automate ingestion with REST APIs and tune parallelism to accelerate ETL loads.

Learn native batch ingestion in Apache Druid, using index parallel for concurrency and index tasks for ingestion, via HTTP APIs to load S3 data with append or overwrite modes.

Learn batch SQL ingestion in Apache Druid using HTTP APIs, with controller and worker tasks in multi-stage workflow, ingesting CSV data from S3 and creating tables with partitioning and clustering.

Access Druid tables from Spark and create a Spark DataFrame with Avatica JDBC. Learn to configure dependencies and validate predicate pushdown and aggregation.

Explore how Apache Druid uses Confluent Schema Registry to validate Avro records from Kafka, ensuring schema compatibility for producers and consumers.

Learn to add the Druid Avro extension, start Druid, and configure a Kafka streaming task with Avro schema and schema registry integration.

Explore how the Druid Hive integration uses Hive sql to translate ansi sql into Druid json queries and push computations to Druid, while Hive handles other steps.

Acquire an AWS EC2 instance, install Java 8 and wget, download and untar Apache Druid, create a log file, and start the Druid server.

Install Kafka on the Druid VM, ingest real-time tweets into Druid via a supervisor task, and query the tweets table in Hive with proper column naming.

Configure a Hive cluster on aws emr with Hive 3.1.2, install the Druid handler jar, set the broker address in Hive site.xml, and restart Hive to enable Druid queries.

Learn to query Druid data from Hive by creating an external Druid table linked to tweets, use explain to inspect the plan, and push queries down to Druid with Druid.query.json.

Explore how the Presto Druid Connector enables querying Druid data and accessing Druid tables, and compare PrestoDB and Trino through setup, ingestion, and PrestoCLI queries.

Install Zookeeper and Kafka with Docker, ingest data from Kafka into Druid via a streaming job, start Druid, and query the Druid table from Presto and Trino.

Install and run Presto on your local machine, connect to a Druid cluster via the druid connector, and query Druid data using the Presto CLI with pushdown enabled.

Connect Druid from Trino by installing and starting the Trino server, using the Trino CLI to query Druid via Avatica JDBC. Configure Druid properties and run SQL with compute pushdown.