【한글자막】 Apache Spark 와 Python으로 빅 데이터 다루기

Apache Spark 3 was released in early 2020 - here's what's new, what's improved, and what's deprecated.

This high-level introduction will help you understand what Spark is for, who's using it, and why it's such a big deal.

Understand the core object of Spark: the Resilient Distributed Dataset (RDD), and how you can use Spark to transform and perform actions upon RDD's.

We'll dissect our original ratings histogram Spark example, and understand exactly how every line of it works!

You'll learn how to use key/value pairs in RDD's, and special operations you can perform on them. To make it real, we'll introduce a new example: computing the average number of friends by age using a fake social network data set.

We'll take another look at our "average number of friends by age" example script, actually run it, and examine the results.

Learn how the filter() operation works on RDD's, and apply this toward finding the minimum temperatures from a real-world weather data set.

We'll look at the minimum temperatures by location example as a whole, and actually run it! Then, you've got an activity: modify this script to find the maximum temperatures instead. This lecture reinforces using filters and key/value RDD's.

Check your results for writing a maximum temperature Spark script to my own.

We'll do the standard "count the number of occurrences of each word in a book" exercise here, and review the differences between map() and flatmap() in the process.

You'll learn how to use regular expressions in Python, and use them to improve the results of our word count script.

Finally, we'll learn how to implement countByValue() in a way that returns a new RDD, and sort that RDD to produce our final results for word frequency.

Write your first Spark script on your own! I'll give you the strategy and tips you need to be successful. You're given a fake e-commerce data set, and your task is to find the total amount spent, broken down by customer ID.

Compare your code to my solution for finding the total spent by customer - and take on a new challenge! Modify your script to sort your final results by the amount spent, and find the biggest spender.

Compare your solution to sorting the customers by total amount ordered to mine, and check your results.

We'll cover the concepts of SparkSQL, DataFrames, and DataSets, and why they are so important in Spark 2.0 and above.

We'll dive into a real example, revisiting our fake social network data and analyzing it with DataFrames through a SparkSession object.

Let's revisit our "most popular movie" example, and implement it using a DataFrame instead of with RDD's. DataFrames are the preferred API in Spark 2.0+.

We'll write and run a simple script to find the most-rated movie in the MovieLens data set, which we'll build upon later.

You'll learn how to use "broadcast variables" in Spark to efficiently distribute large objects to every node your Spark program may be running on, and apply this to looking up movie names in our "most popular movie" script.

We introduce the Marvel superhero social graph data set, and write a Spark job to find the superhero with the most co-occurrences with other heroes in comic books.

Review the source code of our script to discover the most popular superhero, run it, and reveal the answer!

We'll introduce the Breadth-First Search (BFS) algorithm, and how we can use it to discover degrees of separation between superheroes.

We'll learn how to turn breadth-first search into a Spark problem, and craft our strategy for writing the code. Along the way, we'll cover Spark accumulators and how we can use them to signal our driver script when it's done.

We'll get our hands on the code to actually implement breadth-first search, and run it to discover the degrees of separation between any two superheroes!

Learn one technique for finding similar movies based on the MovieLens rating data, and how we can frame it as a Spark problem. We'll also introduce the importance of using cache() or persist() on rdd's that will have more than one action performed on them.

We'll review the code for finding similar movies in Spark with the MovieLens ratings data, run it on every available core of your desktop computer, and review the results.

Get your hands dirty! I'll give you some ideas on improving the quality of your similar movie results - go try some out, and mess around with our movie similarity code.

Learn how Amazon's Elastic MapReduce makes it easy to rent time on your very own Spark cluster, running on top of Hadoop YARN

Learn how to set up your AWS account, create a key pair for logging into your Spark / Hadoop cluster, and set up PuTTY to connect to your instances from a Windows desktop.

We'll see what needs to be done to our Movie Similarities script in order to get it to run successfully with one million ratings, on a cluster, by introducing the partitionBy() function.

We'll study the code of our modified movie similarities script, and get it ready to run on a cluster.

We'll launch a Hadoop cluster with Spark using Amazon's Elastic MapReduce service, and kick off our script to produce similar movies to Star Wars given one million movie ratings.

We'll look at our results from similar movies from one million ratings, and discuss them.

We'll look at the Spark console UI and the information it offers to help understand how to diagnose problems and optimize your large Spark jobs.

I'll share some more troubleshooting tips when running Spark on a cluster, and talk about how to manage dependencies your code may have.

We'll briefly cover the capabilities of Spark's MLLib machine learning library, and how it can help you solve data mining, machine learning, and statistical problems you may encounter. We'll go into more depth on MLLib's Alternating Least Squares (ALS) recommendation engine, and how we can use it to produce movie recommendations with the MovieLens data set.

We'll run MLLib's Alternating Least Squares recommender system on the MovieLens 100K dataset.

We'll finish running Alternating Least Squares recommendations on the MovieLens ratings data set using MLLib, and evaluate the results.

DataFrames are the preferred API for MLLib in Spark 2.0+. Let's look at an example of using linear regression with DataFrames.

An overview of how Spark Streaming lets you process continual streams on input data and aggregate it over time, and how GraphX lets you compute properties of networks.

We'll run an example of Spark structured streaming in Python to keep track of status code counts in a directory that receives Apache access logs.

GraphX isn't currently supported in Python, but you should at least know what it is.

Some suggested resources for learning more about Apache Spark, and data mining and machine learning in general.