Taming Big Data with Apache Spark and Python - Hands On!
4.5 (8,269 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
46,700 students enrolled

Taming Big Data with Apache Spark and Python - Hands On!

Dive right in with 15+ hands-on examples of analyzing large data sets with Apache Spark, on your desktop or on Hadoop!
4.5 (8,269 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
46,700 students enrolled
Last updated 3/2020
English [Auto-generated], French [Auto-generated], 5 more
  • German [Auto-generated]
  • Italian [Auto-generated]
  • Polish [Auto-generated]
  • Portuguese [Auto-generated]
  • Spanish [Auto-generated]
Current price: $104.99 Original price: $149.99 Discount: 30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 5.5 hours on-demand video
  • 1 article
  • 7 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Use DataFrames and Structured Streaming in Spark 3
  • Frame big data analysis problems as Spark problems
  • Use Amazon's Elastic MapReduce service to run your job on a cluster with Hadoop YARN
  • Install and run Apache Spark on a desktop computer or on a cluster
  • Use Spark's Resilient Distributed Datasets to process and analyze large data sets across many CPU's
  • Implement iterative algorithms such as breadth-first-search using Spark
  • Use the MLLib machine learning library to answer common data mining questions
  • Understand how Spark SQL lets you work with structured data
  • Understand how Spark Streaming lets your process continuous streams of data in real time
  • Tune and troubleshoot large jobs running on a cluster
  • Share information between nodes on a Spark cluster using broadcast variables and accumulators
  • Understand how the GraphX library helps with network analysis problems
Course content
Expand all 51 lectures 05:33:02
+ Getting Started with Spark
6 lectures 30:06

Meet your instructor, and we'll review what this course will cover and what you need to get started.

Preview 01:46

How to find the scripts and data associated with the lectures in this course.

How to Use This Course
Udemy 101: Getting the Most From This Course

We'll install Anaconda, a JDK, and Apache Spark on your Windows system. When we're done, we'll run a simple little Spark script on your desktop to test it out!

[Activity]Getting Set Up: Installing Python, a JDK, Spark, and its Dependencies.

Before we can analyze data with Spark, we need some data to analyze! Let's install the MovieLens dataset of movie ratings, which we'll use throughout the course.

[Activity] Installing the MovieLens Movie Rating Dataset

We'll run a simple Spark script using Python, and analyze the 100,000 movie ratings you installed in the previous lecture. What is the breakdown of the rating scores in this data set? You'll find it's easy to find out!

Preview 06:12
+ Spark Basics and Simple Examples
15 lectures 01:53:11

Apache Spark 3 was released in early 2020 - here's what's new, what's improved, and what's deprecated.

Preview 06:48

This high-level introduction will help you understand what Spark is for, who's using it, and why it's such a big deal.

Introduction to Spark

Understand the core object of Spark: the Resilient Distributed Dataset (RDD), and how you can use Spark to transform and perform actions upon RDD's.

The Resilient Distributed Dataset (RDD)

We'll dissect our original ratings histogram Spark example, and understand exactly how every line of it works!

Ratings Histogram Walkthrough

You'll learn how to use key/value pairs in RDD's, and special operations you can perform on them. To make it real, we'll introduce a new example: computing the average number of friends by age using a fake social network data set.

Preview 16:08

We'll take another look at our "average number of friends by age" example script, actually run it, and examine the results.

Preview 05:40

Learn how the filter() operation works on RDD's, and apply this toward finding the minimum temperatures from a real-world weather data set.

Filtering RDD's, and the Minimum Temperature by Location Example

We'll look at the minimum temperatures by location example as a whole, and actually run it! Then, you've got an activity: modify this script to find the maximum temperatures instead. This lecture reinforces using filters and key/value RDD's.

[Activity]Running the Minimum Temperature Example, and Modifying it for Maximums

Check your results for writing a maximum temperature Spark script to my own.

[Activity] Running the Maximum Temperature by Location Example

We'll do the standard "count the number of occurrences of each word in a book" exercise here, and review the differences between map() and flatmap() in the process.

[Activity] Counting Word Occurrences using flatmap()

You'll learn how to use regular expressions in Python, and use them to improve the results of our word count script.

[Activity] Improving the Word Count Script with Regular Expressions

Finally, we'll learn how to implement countByValue() in a way that returns a new RDD, and sort that RDD to produce our final results for word frequency.

Preview 07:46

Write your first Spark script on your own! I'll give you the strategy and tips you need to be successful. You're given a fake e-commerce data set, and your task is to find the total amount spent, broken down by customer ID.

[Exercise] Find the Total Amount Spent by Customer

Compare your code to my solution for finding the total spent by customer - and take on a new challenge! Modify your script to sort your final results by the amount spent, and find the biggest spender.

[Excercise] Check your Results, and Now Sort them by Total Amount Spent.

Compare your solution to sorting the customers by total amount ordered to mine, and check your results.

Check Your Sorted Implementation and Results Against Mine.
+ Advanced Examples of Spark Programs
10 lectures 01:13:13

We'll write and run a simple script to find the most-rated movie in the MovieLens data set, which we'll build upon later.

[Activity] Find the Most Popular Movie

You'll learn how to use "broadcast variables" in Spark to efficiently distribute large objects to every node your Spark program may be running on, and apply this to looking up movie names in our "most popular movie" script.

[Activity] Use Broadcast Variables to Display Movie Names Instead of ID Numbers

We introduce the Marvel superhero social graph data set, and write a Spark job to find the superhero with the most co-occurrences with other heroes in comic books.

Preview 04:29

Review the source code of our script to discover the most popular superhero, run it, and reveal the answer!

[Activity] Run the Script - Discover Who the Most Popular Superhero is!

We'll introduce the Breadth-First Search (BFS) algorithm, and how we can use it to discover degrees of separation between superheroes.

Superhero Degrees of Separation: Introducing Breadth-First Search

We'll learn how to turn breadth-first search into a Spark problem, and craft our strategy for writing the code. Along the way, we'll cover Spark accumulators and how we can use them to signal our driver script when it's done.

Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark

We'll get our hands on the code to actually implement breadth-first search, and run it to discover the degrees of separation between any two superheroes!

[Activity] Superhero Degrees of Separation: Review the Code and Run it

Learn one technique for finding similar movies based on the MovieLens rating data, and how we can frame it as a Spark problem. We'll also introduce the importance of using cache() or persist() on rdd's that will have more than one action performed on them.

Item-Based Collaborative Filtering in Spark, cache(), and persist()

We'll review the code for finding similar movies in Spark with the MovieLens ratings data, run it on every available core of your desktop computer, and review the results.

Preview 10:55

Get your hands dirty! I'll give you some ideas on improving the quality of your similar movie results - go try some out, and mess around with our movie similarity code.

[Exercise] Improve the Quality of Similar Movies
+ Running Spark on a Cluster
8 lectures 49:19

Learn how Amazon's Elastic MapReduce makes it easy to rent time on your very own Spark cluster, running on top of Hadoop YARN

Introducing Elastic MapReduce

Learn how to set up your AWS account, create a key pair for logging into your Spark / Hadoop cluster, and set up PuTTY to connect to your instances from a Windows desktop.

[Activity] Setting up your AWS / Elastic MapReduce Account and Setting Up PuTTY

We'll see what needs to be done to our Movie Similarities script in order to get it to run successfully with one million ratings, on a cluster, by introducing the partitionBy() function.


We'll study the code of our modified movie similarities script, and get it ready to run on a cluster.

Create Similar Movies from One Million Ratings - Part 1

We'll launch a Hadoop cluster with Spark using Amazon's Elastic MapReduce service, and kick off our script to produce similar movies to Star Wars given one million movie ratings.

Preview 11:26

We'll look at our results from similar movies from one million ratings, and discuss them.

Create Similar Movies from One Million Ratings - Part 3

We'll look at the Spark console UI and the information it offers to help understand how to diagnose problems and optimize your large Spark jobs.

Troubleshooting Spark on a Cluster

I'll share some more troubleshooting tips when running Spark on a cluster, and talk about how to manage dependencies your code may have.

More Troubleshooting, and Managing Dependencies
+ SparkSQL, DataFrames, and DataSets
3 lectures 20:16

We'll cover the concepts of SparkSQL, DataFrames, and DataSets, and why they are so important in Spark 2.0 and above.

Introducing SparkSQL

We'll dive into a real example, revisiting our fake social network data and analyzing it with DataFrames through a SparkSession object.

Executing SQL commands and SQL-style functions on a DataFrame

Let's revisit our "most popular movie" example, and implement it using a DataFrame instead of with RDD's. DataFrames are the preferred API in Spark 2.0+.

Preview 05:52
+ Other Spark Technologies and Libraries
7 lectures 42:30

We'll briefly cover the capabilities of Spark's MLLib machine learning library, and how it can help you solve data mining, machine learning, and statistical problems you may encounter. We'll go into more depth on MLLib's Alternating Least Squares (ALS) recommendation engine, and how we can use it to produce movie recommendations with the MovieLens data set.

Introducing MLLib

We'll run MLLib's Alternating Least Squares recommender system on the MovieLens 100K dataset.

[Activity] Using MLLib to Produce Movie Recommendations

We'll finish running Alternating Least Squares recommendations on the MovieLens ratings data set using MLLib, and evaluate the results.

Analyzing the ALS Recommendations Results

DataFrames are the preferred API for MLLib in Spark 2.0+. Let's look at an example of using linear regression with DataFrames.

Preview 07:31

An overview of how Spark Streaming lets you process continual streams on input data and aggregate it over time, and how GraphX lets you compute properties of networks.

Spark Streaming

We'll run an example of Spark structured streaming in Python to keep track of status code counts in a directory that receives Apache access logs.

Preview 08:47

GraphX isn't currently supported in Python, but you should at least know what it is.

+ You Made It! Where to Go from Here.
2 lectures 04:27

Some suggested resources for learning more about Apache Spark, and data mining and machine learning in general.

Learning More about Spark and Data Science
Bonus Lecture: More courses to explore!
  • Access to a personal computer. This course uses Windows, but the sample code will work fine on Linux as well.
  • Some prior programming or scripting experience. Python experience will help a lot, but you can pick it up as we go.

New! Updated for Spark 3 and with a hands-on structured streaming example.

“Big data" analysis is a hot and highly valuable skill – and this course will teach you the hottest technology in big data: Apache Spark. Employers including Amazon, EBay, NASA JPL, and Yahoo all use Spark to quickly extract meaning from massive data sets across a fault-tolerant Hadoop cluster. You'll learn those same techniques, using your own Windows system right at home. It's easier than you might think.

Learn and master the art of framing data analysis problems as Spark problems through over 15 hands-on examples, and then scale them up to run on cloud computing services in this course. You'll be learning from an ex-engineer and senior manager from Amazon and IMDb.

  • Learn the concepts of Spark's Resilient Distributed Datastores

  • Develop and run Spark jobs quickly using Python

  • Translate complex analysis problems into iterative or multi-stage Spark scripts

  • Scale up to larger data sets using Amazon's Elastic MapReduce service

  • Understand how Hadoop YARN distributes Spark across computing clusters

  • Learn about other Spark technologies, like Spark SQL, Spark Streaming, and GraphX

By the end of this course, you'll be running code that analyzes gigabytes worth of information – in the cloud – in a matter of minutes. 

This course uses the familiar Python programming language; if you'd rather use Scala to get the best performance out of Spark, see my "Apache Spark with Scala - Hands On with Big Data" course instead.

We'll have some fun along the way. You'll get warmed up with some simple examples of using Spark to analyze movie ratings data and text in a book. Once you've got the basics under your belt, we'll move to some more complex and interesting tasks. We'll use a million movie ratings to find movies that are similar to each other, and you might even discover some new movies you might like in the process! We'll analyze a social graph of superheroes, and learn who the most “popular" superhero is – and develop a system to find “degrees of separation" between superheroes. Are all Marvel superheroes within a few degrees of being connected to The Incredible Hulk? You'll find the answer.

This course is very hands-on; you'll spend most of your time following along with the instructor as we write, analyze, and run real code together – both on your own system, and in the cloud using Amazon's Elastic MapReduce service. 5 hours of video content is included, with over 15 real examples of increasing complexity you can build, run and study yourself. Move through them at your own pace, on your own schedule. The course wraps up with an overview of other Spark-based technologies, including Spark SQL, Spark Streaming, and GraphX.

Wrangling big data with Apache Spark is an important skill in today's technical world. Enroll now!

  • " I studied "Taming Big Data with Apache Spark and Python" with Frank Kane, and helped me build a great platform for Big Data as a Service for my company. I recommend the course!  " - Cleuton Sampaio De Melo Jr.

Who this course is for:
  • People with some software development background who want to learn the hottest technology in big data analysis will want to check this out. This course focuses on Spark from a software development standpoint; we introduce some machine learning and data mining concepts along the way, but that's not the focus. If you want to learn how to use Spark to carve up huge datasets and extract meaning from them, then this course is for you.
  • If you've never written a computer program or a script before, this course isn't for you - yet. I suggest starting with a Python course first, if programming is new to you.
  • If your software development job involves, or will involve, processing large amounts of data, you need to know about Spark.
  • If you're training for a new career in data science or big data, Spark is an important part of it.