Spark Performance Tuning for Data Engineers: Part1 - Storage

Explore Spark's master–slave architecture and how a driver coordinates executors across a cluster. Understand the job flow from submission to final results, including resources, tasks, data partitions, and progress reporting.

Understand how Spark executes jobs: actions define jobs, driver creates logical plans, breaks into stages with wide and narrow transformations, and uses shuffle, tasks, and partitions.

Explore spark ui to see how partitions map to cores and memory, and observe jobs, stages, and dag flow during read, filter, and write operations.

Explore how spark builds physical plans and DAGs for joins and group by operations, using hash partitioning, shuffle, and sort-merge join, with partial and final aggregates for efficient data processing.

Discover why reusing a data frame causes repeated storage reads in Spark, and how explicit in-memory caching enables faster, repeated queries across multiple actions.

Eliminate unused columns when caching data frames to reduce memory and read data, improving first-query latency and subsequent cached-query performance on parquet datasets.

Partition data by year and month to enable predicate pushdown and reduce storage reads. This row elimination approach enhances linear scalability for filtered queries by reading only the needed data.

Explore the directory scan problem in Spark when reading partitioned data and hierarchical year month day hour directories, and learn to mitigate with low-cardinality partitions, z-order indexing, and bucketing.

Understand the haystack query problem and learn to use partitioning and z-order indexing to enable predicate pushdown, partition pruning, and faster day/hour filters.