Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Scalable programming with Scala and Spark

Use Scala and Spark for data analysis, machine learning and analytics

Created byLoony Corn

Last updated 8/2019

English

What you'll learn

Use Spark for a variety of analytics and Machine Learning tasks
Understand functional programming constructs in Scala
Implement complex algorithms like PageRank or Music Recommendations
Work with a variety of datasets from Airline delays to Twitter, Web graphs, Social networks and Product Ratings
Use all the different features and libraries of Spark : RDDs, Dataframes, Spark SQL, MLlib, Spark Streaming and GraphX
Write code in Scala REPL environments and build Scala applications with an IDE

Course content

12 sections • 55 lectures • 9h 1m total length

You, This Course and Us2:16
Course Materials0:04
Installing Scala and Hello World9:43
Install Scala and use it both in the shell and with an IDE

What does Donald Rumsfeld have to do with data analysis?8:46
He has a great categorization for insights in data, really!

There is a profound truth in here which data scientists and analysts have known for years.
Why is Spark so cool?12:23
Explore, investigate and find patterns in data. Build fully fledged, scalable productions system. All using the same environment.
An introduction to RDDs - Resilient Distributed Datasets9:39
RDDs are pretty magical, they are the core programming abstraction in Spark
Built-in libraries for Spark15:38
Spark is even more powerful because of the packages that come with it. Spark SQL, Spark Streaming, MLlib and GraphX.
Installing Spark11:44
Let's get started by installing Spark. We'll also configure Spark to work with IPython Notebook
The Spark Shell6:55
Start munging data using the Spark REPL environment.
See it in Action : Munging Airlines Data with Spark3:44
We've learnt a little bit about how Spark and RDDs work. Let's see it in action!
Transformations and Actions17:06
Operations on data, transform data to extract information and then retrieve results.

RDD Characteristics: Partitions and Immutability12:35
RDDs are very intuitive to use. What are some of the characteristics that make RDDs performant, resilient and efficient?
RDD Characteristics: Lineage, RDDs know where they came from6:05
Lazy evaluation of RDDs is possible because RDDs can reconstruct themselves. They know where they came from.
What can you do with RDDs?11:08
A quick overview of all operations and transformations on RDDs
Create your first RDD from a file14:54
Parse a CSV file, transform is using the map() operation, create Flight objects on the fly.
Average distance travelled by a flight using map() and reduce() operations6:59
Use the flights dataset to get interesting insights.
Get delayed flights using filter(), cache data using persist()6:10
Cache RDDs in memory to optimize operations using persist()
Average flight delay in one-step using aggregate()12:21
Use the aggregate() operation to calculate average flight delays in one step. Much more compact than map() and reduce().
Frequency histogram of delays using countByValue()2:10
This is surprisingly simple!

Special Transformations and Actions14:45
Pair RDDs are special types of RDDs where every record is a key value pair. All normal actions and transformations apply to these in addition to some special ones.
Average delay per airport, use reduceByKey(), mapValues() and join()13:35
Pair RDDs are useful to get information on a per-key basis. Sales per city, delays per airport etc.
Average delay per airport in one step using combineByKey()8:23
Instead of 3 steps use just one to get the average delay per airport.
Get the top airports by delay using sortBy()2:51
Sort RDDs easily
Lookup airport descriptions using lookup(), collectAsMap(), broadcast()10:56
Looking up airport descriptions in a pair RDD can be done in many ways, understand how each work.

Get information from individual processing nodes using accumulators9:25
Accumulators are special variables which allow the main driver program to collect information from nodes on which the actual processing takes place.
Long running programs using spark-submit7:11
Spark is more than just the Read-Evaluate-Print Loop environment, it can run long running programs as well.
Spark-Submit with Scala - A demo6:09
Behind the scenes: What happens when a Spark script runs?14:30
How does Spark submit jobs for distributed processing? How does the scheduler work? What does the cluster manager do? All this and more in this behind the scenes.
Running MapReduce operations10:53
MapReduce is a powerful paradigm for distributed processing. Many tasks lend themselves well to this model and Spark has transformations which deal with this beautifully.

Collaborative filtering algorithms12:19
This is a family of algorithms which give recommendations based on user data and user preferences.
Latent Factor Analysis with the Alternating Least Squares method11:39
One type of collaborative filtering algorithm is latent factor analysis. There is some math here but don't worry, MLlib takes care of all this for you.
Music recommendations using the Audioscrobbler dataset5:38
Let's write a recommendation engine for music services
Implement code in Spark using MLlib14:45
The code in Spark is surprisingly simple.

Requirements

All examples work with or without Hadoop. If you would like to use Spark with Hadoop, you'll need to have Hadoop installed (either in pseudo-distributed or cluster mode).
The course assumes experience with one of the popular object-oriented programming languages like Java/C++

Description

Taught by a 4 person team including 2 Stanford-educated, ex-Googlers and 2 ex-Flipkart Lead Analysts. This team has decades of practical experience in working with Java and with billions of rows of data.

Get your data to fly using Spark and Scala for analytics, machine learning and data science

Let’s parse that.

What's Spark? If you are an analyst or a data scientist, you're used to having multiple systems for working with data. SQL, Python, R, Java, etc. With Spark, you have a single engine where you can explore and play with large amounts of data, run machine learning algorithms and then use the same system to productionize your code.

Scala: Scala is a general purpose programming language - like Java or C++. It's functional programming nature and the availability of a REPL environment make it particularly suited for a distributed computing framework like Spark.

Analytics: Using Spark and Scala you can analyze and explore your data in an interactive environment with fast feedback. The course will show how to leverage the power of RDDs and Dataframes to manipulate data with ease.

Machine Learning and Data Science : Spark's core functionality and built-in libraries make it easy to implement complex algorithms like Recommendations with very few lines of code. We'll cover a variety of datasets and algorithms including PageRank, MapReduce and Graph datasets.

What's Covered:

Scala Programming Constructs: Classes, Traits, First Class Functions, Closures, Currying, Case Classes

Lot's of cool stuff ..

Music Recommendations using Alternating Least Squares and the Audioscrobbler dataset
Dataframes and Spark SQL to work with Twitter data
Using the PageRank algorithm with Google web graph dataset
Using Spark Streaming for stream processing
Working with graph data using the Marvel Social network dataset

.. and of course all the Spark basic and advanced features:

Resilient Distributed Datasets, Transformations (map, filter, flatMap), Actions (reduce, aggregate)
Pair RDDs , reduceByKey, combineByKey
Broadcast and Accumulator variables
Spark for MapReduce
The Java API for Spark
Spark SQL, Spark Streaming, MLlib and GraphX

Who this course is for:

Yep! Engineers who want to use a distributed computing engine for batch or stream processing or both
Yep! Analysts who want to leverage Spark for analyzing interesting datasets
Yep! Data Scientists who want a single engine for analyzing and modelling data as well as productionizing it.

Scalable programming with Scala and Spark

What you'll learn

Explore related topics

Course content

You, This Course and Us3 lectures • 12min

Introduction to Spark8 lectures • 1hr 26min

Resilient Distributed Datasets8 lectures • 1hr 12min

Advanced RDDs: Pair Resilient Distributed Datasets5 lectures • 51min

Advanced Spark: Accumulators, Spark Submit, MapReduce , Behind The Scenes5 lectures • 48min

PageRank: Ranking Search Results4 lectures • 39min

Spark SQL1 lecture • 16min

MLlib in Spark: Build a recommendations engine4 lectures • 44min

Spark Streaming3 lectures • 28min

Graph Libraries1 lecture • 15min

Requirements

Description

Who this course is for: