Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

From 0 to 1 : Spark for Data Science with Python

Name: From 0 to 1 : Spark for Data Science with Python
Rating: 4.2 (779 reviews)

Get your data to fly using Spark for analytics, machine learning and data science

Created byLoony Corn

Last updated 2/2018

English

What you'll learn

Use Spark for a variety of analytics and Machine Learning tasks
Implement complex algorithms like PageRank or Music Recommendations
Work with a variety of datasets from Airline delays to Twitter, Web graphs, Social networks and Product Ratings
Use all the different features and libraries of Spark : RDDs, Dataframes, Spark SQL, MLlib, Spark Streaming and GraphX

Course content

11 sections • 54 lectures • 8h 18m total length

You, This Course and Us2:15
Explore Spark as a single system to study large data, run machine learning, and make predictions, using map, reduce, and aggregate with RDD workflows, reduceByKey and combineByKey, plus Java API.
Course Materials0:04

What does Donald Rumsfeld have to do with data analysis?8:45
He has a great categorization for insights in data, really!

There is a profound truth in here which data scientists and analysts have known for years.
Why is Spark so cool?12:22
Explore, investigate and find patterns in data. Build fully fledged, scalable productions system. All using the same environment.
An introduction to RDDs - Resilient Distributed Datasets9:38
RDDs are pretty magical, they are the core programming abstraction in Spark
Built-in libraries for Spark15:37
Spark is even more powerful because of the packages that come with it. Spark SQL, Spark Streaming, MLlib and GraphX.
Installing Spark6:42
Let's get started by installing Spark. We'll also configure Spark to work with IPython Notebook
The PySpark Shell4:49
Start munging data using the PySpark REPL environment.
Transformations and Actions13:32
Operations on data, transform data to extract information and then retrieve results.
See it in Action : Munging Airlines Data with PySpark - I10:13
We've learnt a little bit about how Spark and RDDs work. Let's see it in action!
[For Linux/Mac OS Shell Newbies] Path and other Environment Variables8:25
If you are unfamiliar with softwares that require working with a shell/command line environment, this video will be helpful for you. It explains how to update the PATH environment variable, which is needed to set up most Linux/Mac shell based softwares.

RDD Characteristics: Partitions and Immutability12:34
RDDs are very intuitive to use. What are some of the characteristics that make RDDs performant, resilient and efficient?
RDD Characteristics: Lineage, RDDs know where they came from6:05
Lazy evaluation of RDDs is possible because RDDs can reconstruct themselves. They know where they came from.
What can you do with RDDs?11:08
A quick overview of all operations and transformations on RDDs
Create your first RDD from a file16:10
Parse a CSV file, transform is using the map() operation, create Flight objects on the fly.
Average distance travelled by a flight using map() and reduce() operations5:49
Use the flights dataset to get interesting insights.
Get delayed flights using filter(), cache data using persist()5:23
Cache RDDs in memory to optimize operations using persist()
Average flight delay in one-step using aggregate()15:10
Use the aggregate() operation to calculate average flight delays in one step. Much more compact than map() and reduce().
Frequency histogram of delays using countByValue()3:25
This is surprisingly simple!
See it in Action : Analyzing Airlines Data with PySpark - II6:25
See all of the RDD operations in action. Using map, reduce, aggregate to analyze Airline data.

Special Transformations and Actions14:44
Pair RDDs are special types of RDDs where every record is a key value pair. All normal actions and transformations apply to these in addition to some special ones.
Average delay per airport, use reduceByKey(), mapValues() and join()18:10
Pair RDDs are useful to get information on a per-key basis. Sales per city, delays per airport etc.
Average delay per airport in one step using combineByKey()11:52
Instead of 3 steps use just one to get the average delay per airport.
Get the top airports by delay using sortBy()4:33
Sort RDDs easily
Lookup airport descriptions using lookup(), collectAsMap(), broadcast()14:01
Looking up airport descriptions in a pair RDD can be done in many ways, understand how each work.
See it in Action : Analyzing Airlines Data with PySpark - III4:58
Analyze airlines data with the help of Pair RDDs.

Get information from individual processing nodes using accumulators13:35
Accumulators are special variables which allow the main driver program to collect information from nodes on which the actual processing takes place.
See it in Action : Using an Accumulator variable2:40
Explore using an accumulator to count log lines containing a target word while processing logs with a map function, loading logs into memory, and saving results to a text file.
Long running programs using spark-submit5:58
Spark is more than just the Read-Evaluate-Print Loop environment, it can run long running programs as well.
See it in Action : Running a Python script with Spark-Submit3:58
Submit a Python script to Spark as a standalone job with spark-submit, initializing a Spark context to connect to the cluster and count log lines.
Behind the scenes: What happens when a Spark script runs?14:30
How does Spark submit jobs for distributed processing? How does the scheduler work? What does the cluster manager do? All this and more in this behind the scenes.
Running MapReduce operations13:44
MapReduce is a powerful paradigm for distributed processing. Many tasks lend themselves well to this model and Spark has transformations which deal with this beautifully.
See it in Action : MapReduce with Spark2:05
Demonstrate MapReduce with Spark by loading a text file, flatMap to extract words, map to (word, 1), reduce by key to count, and collect to print a dictionary.

The Java API and Function objects15:58
Spark works with Java as well. If that is your language of choice then you have reason to rejoice.
Pair RDDs in Java4:49
Pair RDDs in Java have to be created explicitly, a tuple RDD is not automatically a Pair RDD
Running Java code3:49
Using spark-submit with Java code.
Installing Maven2:20
Maven is a prerequisite to compiling and building your Java JARs for Spark.
See it in Action : Running a Spark Job with Java5:08
Learn how to set up a Maven Java project, declare groupId and artifactId, add Spark dependencies, compile and package, then run the Spark job with spark-submit.

What is PageRank?16:44
Explore the PageRank algorithm, modeling the web as a directed graph where pages earn rank from linked pages, iteratively updated with damping to converge on global importance.
The PageRank algorithm6:15
Implement the PageRank algorithm in Spark by building the web graph from the Google 2002 data set, initializing ranks, joining links, distributing ranks, applying damping, and iterating to convergence.
Implement PageRank in Spark12:01
This will be way simpler than the explanation.
Join optimization in PageRank using Custom Partitioning7:27
Optimize the algorithm by making joins more performant
See it Action : The PageRank algorithm using Spark3:46
Learn how to implement PageRank in Spark on a Stanford web graph dataset, iterating rank updates across neighbors, with partitioning, persisting, and extracting top 10 pages.

Collaborative filtering algorithms12:19
This is a family of algorithms which give recommendations based on user data and user preferences.
Latent Factor Analysis with the Alternating Least Squares method11:39
One type of collaborative filtering algorithm is latent factor analysis. There is some math here but don't worry, MLlib takes care of all this for you.
Music recommendations using the Audioscrobbler dataset7:51
Let's write a recommendation engine for music services
Implement code in Spark using MLlib16:05
The code in Spark is surprisingly simple.

Introduction to streaming9:55
Spark can process streaming data in near real time using DStreams.
Implement stream processing in Spark using Dstreams10:54
A script to parse logs in real time
Stateful transformations using sliding windows9:26
Stateful transformations allow cumulative results across a stream using a sliding window.
See it in Action : Spark Streaming4:17
See a live Spark streaming demo that sets up a streaming server, processes text to count 'error', and displays results with a 30-second window and a 10-second slide.

Requirements

The course assumes knowledge of Python. You can write Python code directly in the PySpark shell. If you already have IPython Notebook installed, we'll show you how to configure it for Spark
For the Java section, we assume basic knowledge of Java. An IDE which supports Maven, like IntelliJ IDEA/Eclipse would be helpful
All examples work with or without Hadoop. If you would like to use Spark with Hadoop, you'll need to have Hadoop installed (either in pseudo-distributed or cluster mode).

Description

Taught by a 4 person team including 2 Stanford-educated, ex-Googlers and 2 ex-Flipkart Lead Analysts. This team has decades of practical experience in working with Java and with billions of rows of data.

Get your data to fly using Spark for analytics, machine learning and data science

Let’s parse that.

What's Spark? If you are an analyst or a data scientist, you're used to having multiple systems for working with data. SQL, Python, R, Java, etc. With Spark, you have a single engine where you can explore and play with large amounts of data, run machine learning algorithms and then use the same system to productionize your code.

Analytics: Using Spark and Python you can analyze and explore your data in an interactive environment with fast feedback. The course will show how to leverage the power of RDDs and Dataframes to manipulate data with ease.

Machine Learning and Data Science : Spark's core functionality and built-in libraries make it easy to implement complex algorithms like Recommendations with very few lines of code. We'll cover a variety of datasets and algorithms including PageRank, MapReduce and Graph datasets.

What's Covered:

Lot's of cool stuff ..

Music Recommendations using Alternating Least Squares and the Audioscrobbler dataset
Dataframes and Spark SQL to work with Twitter data
Using the PageRank algorithm with Google web graph dataset
Using Spark Streaming for stream processing
Working with graph data using the Marvel Social network dataset

.. and of course all the Spark basic and advanced features:

Resilient Distributed Datasets, Transformations (map, filter, flatMap), Actions (reduce, aggregate)
Pair RDDs , reduceByKey, combineByKey
Broadcast and Accumulator variables
Spark for MapReduce
The Java API for Spark
Spark SQL, Spark Streaming, MLlib and GraphFrames (GraphX for Python)

Who this course is for:

Yep! Analysts who want to leverage Spark for analyzing interesting datasets
Yep! Data Scientists who want a single engine for analyzing and modelling data as well as productionizing it.
Yep! Engineers who want to use a distributed computing engine for batch or stream processing or both

From 0 to 1 : Spark for Data Science with Python

What you'll learn

Explore related topics

Course content

You, This Course and Us2 lectures • 2min

Introduction to Spark9 lectures • 1hr 30min

Resilient Distributed Datasets9 lectures • 1hr 22min

Advanced RDDs: Pair Resilient Distributed Datasets6 lectures • 1hr 8min

Advanced Spark: Accumulators, Spark Submit, MapReduce , Behind The Scenes7 lectures • 57min

Java and Spark5 lectures • 32min

PageRank: Ranking Search Results5 lectures • 46min

Spark SQL2 lectures • 21min

MLlib in Spark: Build a recommendations engine4 lectures • 48min

Spark Streaming4 lectures • 35min

Requirements

Description

Who this course is for: