Apache Spark - Beyond Basics and Cracking Job Interviews

Name: Apache Spark - Beyond Basics and Cracking Job Interviews
Rating: 4.6 (4995 reviews)

Learn Spark Internals, Architecture, Advanced Concepts to Prepare for Databricks Certification Exam and Job Interviews

Bestseller

Created byPrashant Kumar Pandey, Learning Journal

Last updated 7/2024

English

What you'll learn

Apache Spark 3 Advanced Topics and Concept
Deep Dive in to Spark 3 Architecture and Memory Management
Learn AQE, DPP, Broadcast, Accumulators, and Multithreading in Spark 3
Common Job Interview Questions and Answers

Course content

3 sections • 25 lectures • 4h 6m total length

Course Introduction2:12

Spark Cluster Runtime Architecture8:16
Spark Submit and Important Options4:01
Learn how to submit spark apps with spark-submit and configure master, deploy mode, conf, and driver/executor resources on a cluster.
Deploy Modes - Client and Cluster mode4:57
Spark Jobs - Stage, Shuffle, Task, Slots15:41
Spark SQL Engine and Query Planning5:24
Lets Practice - Quiz 1
Lets Practice - Quiz 1 Solution Video19:20
Lets Practice - Quiz 2
Lets Practice - Quiz 2 Solution Video14:13

Spark Memory Allocation9:43
Explain how spark driver and executor memory are allocated on a yarn cluster. Detail heap, overhead, and configurations like spark.driver.memory and spark.executor.memoryOverhead that determine total container memory.
Spark Memory Management14:26
Spark Adaptive Query Execution12:37
Spark AQE Dynamic Join Optimization7:40
Handling Data Skew in Spark Joins7:51
Discover how Spark adaptive query execution handles skewed joins by dynamically splitting skewed partitions and enabling skew-join optimization with AQE.
Spark Dynamic Partition Pruning9:31
Learn how dynamic partition pruning in Spark 3.0 speeds queries by pruning partitions via a broadcast dimension table and injecting subqueries into the fact table.
Data Caching in Spark14:27
Repartition and Coalesce8:26
Dataframe Hints4:53
Broadcast Variables6:25
Accumulators7:45
Use Spark accumulators as global mutable counters to track bad records during data fixes, reducing shuffles and ensuring accurate counts via actions.
Speculative Execution5:53
Dynamic Resource Allocation7:43
Spark Schedulers7:28
Unit Testing in Spark20:05
Develop modular Spark applications with a lib/transformations module and two unit-tested functions, load_survey_df and count_by_country, using unittest in local and notebook environments.
Lets Practice - Quiz 3
Lets Practice - Quiz 3 Solution Video15:46
Practice quiz solutions review spark shuffle basics, map side repartitioning, partition aggregation, and the differences between transformations and actions for data skew, coalesce, and repartition scenarios.
Lets Practice - Quiz 4
Lets Practice - Quiz 4 Solution Video11:37
Master Spark optimization techniques through a quiz-driven walkthrough of spark.speculation, memory management with spark.memory.fraction, adaptive query execution, dynamic partition pruning, and join strategy hints.

Requirements

Basic knowledge of Spark programing in Python - PySpark

Description

Are you preparing for Databricks Spark Certification?

Or, you might be preparing for Spark Job Interviews.

Or you are looking for a place where you can learn Spark Architecture, Internals and advanced topics.

You are at the right place.

I designed this course with the following objectives.

Learn Advanced Spark Skills
Prepare for advanced Certification topics
Prepare and crack Spark Job Interviews

This course is not for beginners.

So if you are a beginner and want to learn from scratch, you should take my PySpark - Apache Spark Programming in Python for beginners course.

In this course, you will learn some advanced Spark skills that I have already collected as feedback from various students and professionals. This course also covers some advanced topics and concepts that are asked in the Databricks Spark certification exam. So PySpark - Apache Spark Programming in Python for beginners and Apache Spark - Beyond Basics and Cracking Job Interviews together cover 100% of the Databriks Certified Spark Developer Associate certification curriculum.

Many of the concepts covered in this course are part of the Data Engineering job interviews for Spark Developers.

So this course will also help you crack the Spark Developer and Data Engineer Job interviews.

Looking forward to seeing you on the course.

Keep Learning and Keep Growing.

Who this course is for:

Anyone who wants to learn and develop Advanced Skills in Apache Spark
Those wo are preparing for Job Interviews and want to learn advanced skills

Apache Spark - Beyond Basics and Cracking Job Interviews

What you'll learn

Explore related topics

Course content

Before your begin1 lecture • 2min

Spark Architecture7 lectures • 1hr 12min

Performance and Applied Understanding17 lectures • 2hr 52min

Requirements

Description

Who this course is for: