
Linkedin - www.linkedin.com/in/suprobho-santra
Understand performance tuning aims to boost throughput, lower CPU utilization, and reduce failures. Learn to benchmark measurable, repeatable, and consistent improvements while pinpointing bottlenecks and opportunities.
Benchmark by running baseline and optimized code in a controlled environment, measure matrices such as response time, throughput, and memory, and use noop write to ensure repeatable, consistent results.
Master Spark high level architecture, including a driver and multiple executors managed by a cluster manager. Learn how the driver assigns tasks and processes data partitions across the cluster.
Understand how Spark executes applications by breaking code into jobs, stages, and tasks. Explore how actions create jobs, wide transformations cause shuffles, and partitions drive parallel tasks.
Explore Spark UI fundamentals using a simple notebook, including executors, drivers, memory, partitions, and how stages and jobs map to read, filter, and project operations.
Explore how spark converts a submitted query into an unresolved logical plan, validates it against the catalog, and generates optimized logical and physical plans with catalyst and adaptive query execution.
Explore how Spark builds physical plans and DAGs for repartitioning, hash partitioning, and sort-merge joins on the call type key, including inner joins and group-by aggregation.
Understand data spill in Spark by seeing how large partitions exceed memory and spill to disk, and learn how maintaining moderate partitions prevents spill.
Understand how Spark executor memory is allocated in Databricks, including kernel overhead, node services, and the Spark memory fraction that splits storage and user memory.
This lecture analyzes data spill in a spark job, reveals memory spill from large partitions, and outlines fixes: increase memory per core or use larger vms; optionally reduce partition size.
Increase spark.memory.fraction to 0.8 for about 30 percent more memory to address the data spill problem, and benchmark on a Databricks cluster.
Benchmarking with large VMs increases memory per core but can worsen IO and data spill, degrading query performance. Adjusting input partition size offers a more elegant spill solution.
Address data spill by tuning shuffle partitions rather than adding memory, setting spark.sql.shuffle.partitions toward about 1500 to reduce median partition size toward 100–120 MB and eliminate spill.
Unlock the true potential of Apache Spark by mastering storage-related performance tuning techniques. This hands-on course is packed with real-world scenarios, guided demos, and practical use cases that will help you fine-tune Spark storage strategies for speed, efficiency, and scalability.
This course is perfect for Intermediate Data Engineers & Spark Developers as well as Aspiring Achitects who wants to optimize Spark jobs, reduce resource costs, and ensure fast, reliable performance for large-scale data applications.
What You’ll Learn
1. Understand how Apache Spark handles storage internally: memory vs disk
2. Learn when and how to use Spark caching and persistence effectively
3. Compare and choose the right storage levels: MEMORY_ONLY, MEMORY_AND_DISK, etc.
4. Use real-world examples and hands-on demos to benchmark storage decisions
5. Learn how to monitor storage metrics using the Spark UI
6. Handle memory spills, disk I/O bottlenecks, and storage tuning in cluster environments
7. Apply best practices for storage optimization in cloud and on-prem Spark clusters
Why Take This Course?
100% Hands-on: Focused on practical implementation, not just theory
Designed for Data Engineers, Spark Developers, and Big Data Practitioners
Covers both foundational concepts and advanced tuning techniques
Teaches how to measure performance gains using real metrics
Helps you make cost-efficient decisions for big data storage
Tools & Technologies Covered
Apache Spark (2.x and 3.x)
DataBricks
Spark UI
HDFS, DataLake (for storage scenarios)