Overcoming Common Performance Issues in Apache Spark
What you'll learn
- The three main causes of performance issues in Apache Spark
- How to overcome shuffle induced performance issues in Apache Spark
- How to overcome skew induced performance issues in Apache Spark
- How to overcome spill induced performance issues in Apache Spark
- Apache Spark Programming
Spark is a powerful framework for processing large datasets in parallel. But, with the complex architecture come frequent performance issues.
In my experience, it can be frustrating looking everywhere, trying to find a resource online that is worded in such a way that you fully understand the inner workings of Spark and how to address these issues. So, I created this course!
This is not a code-along course. This course assumes you already know how to code in Spark. Here, we're talking about how you resolve the performance issues that you encounter during your development journey! We will walk through all of the theory & you'll have actionable steps to take to resolve your performance issues.
In this course, we will cover off:
The Apache Spark Architecture
The type of deployment modes in Apache Spark
The structure of jobs in Apache Spark
How to handle the three main performance concerns in Spark
If you don't yet know how to code in Spark, you can join my 60 minute crash course in PySpark, here on Udemy.
Let's get to work understanding why your scripts are not performing as you may hope and resolve your performance issues together. Shuffle, Skew and Spill will be concerns of the past after this course!
Who this course is for:
- Spark developers looking to improve performance of their scripts
Hey guys! I am a data engineer by trade and specialize in Python, SQL, Spark, Hive, MongoDB and more. I've come on Udemy to try and make simple, short crash courses into these technologies as I personally find the longer courses too drawn out & I often lose interest. The idea is to keep it short and sharp!
For loads of advanced Spark, Python & Big Data topics, please visit my website (the button on this page will take you there) - where I talk about scaling up to enterprise grade solutions.