Apache Spark with Scala By Example

Before we jump into Spark with Scala examples, let's presenting a high-level overview of the key concepts you need to know. These fundamentals will be used throughout the rest of this Spark with Scala course.

Key constructs: Resilient Distributed Datasets (RDDs), Transformations, Actions, Spark Driver programs, SparkContext and how applications deployed to a Spark cluster utilize the parallel nature of Spark.

We're going to be running many examples in this next section.  I don't expect you to follow every detail.  Rather, I just want to experience loading external data and run some simple examples of Spark Transformations and Actions. 

To begin the course, let's run some Spark code with Scala from the shell.

I don't expect you to follow all the details of this code. I just want to get us motivated to continue our Spark learning adventure.

In this example, we'll get a glimpse into Spark core concepts such as Resilient Distributed Datasets, Transformations, Actions and Spark drivers from a Scala perspective. Again, I'll fill in all the details of this Scala code in later lectures.

In this section of the Spark with Scala course, we'll set up and verify your Spark with Scala environment. With your own environment in place, you can choose to run the course examples and experiment with the Scala Spark API.

Walk through all steps required to setup Apache Spark on your machine.

We need sample data to run Scala examples in the Spark Console. This lecture will prepare the Apache Spark environment for loading data and confirm the Spark console.

Reference links used in this section of the Spark with Scala course

There are two kinds of Spark functions: Transformations and Actions. Transformations transform an existing RDD into a new, different one. Actions are functions used against RDDs to produce a value.

In this section of the Apache Spark with Scala course, we'll go over a variety of Spark Transformation and Action functions.

This should build your confidence and understanding of how you can apply these functions to your uses cases. It will also create more foundation for us to build upon in your journey of learning Apache Spark with Scala.

What are Spark Transformations? Let's review common Spark Transformation functions through Scala code examples.

We're going to break Apache Spark transformations into groups. In this video, we'll cover some common spark transformations which produce RDDs. These include map, flatMap, filter, etc.

We're going to use a CSV dataset of baby names in New York. As we progress through transformations and actions in this Apache Spark with Scala course, we'll determine more and more results for this sample data set.

So, let's begin with some commonly used Spark transformations.

In part 2 of Spark Transformations, we'll discover spark transformations used when we need to combine, compare and contrast elements in two RDDs. This is something we often have to do when working with datasets. Spark helps compare RDDs through transformation functions union, intersection, distinct, etc.

In part 3 of our focus on Spark Transformation functions were going work with the "key" functions including groupByKey, reduceByKey, aggregateByKey, sortByKey

All these transformations work with key,value pair RDDs, so we will cover the creation of PairRDDs as well.

We'll continue to use the baby_names.csv file used in Part 1 and Part 2 of Spark Transformations

Run and review common Spark actions. You have already seen many Spark action examples before this lecture, so we will go quickly to review.

Spark Actions produce values back to the Spark Driver program. Also, recall that Action functions called against RDD cause a previously lazy RDD to be evaluated. So, in the real world when working with large datasets, we need to be careful when triggering RDDs to be evaluated through Spark actions.

This video shows commonly used Spark Actions.

Links to conveniently download the Spark source code examples presented in this section of the course.  Also, links to the latest programming guides for SparkTransformations and Actions is included. 

Clusters allow Spark to processes huge volumes of data by distributing the workload across multiple nodes. This is also referred to as "running in parallel" or "horizontal scaling"

A cluster manager is required to Spark on a cluster. Spark supports 3 types of cluster managers including Apache YARN, Apache Mesos and an internal cluster manager distributed with Spark called Standalone.

Let's run a Spark Standalone cluster within your environment. We'll start a Spark Master and one Spark worker. We'll introduce the Spark UI web console.

Setup, compile and package a Scala Spark program using `sbt`.  `sbt` is short for "simple build tool" and is most often used in Scala based projects. 

This is easy example to ensure you're ready for more advanced build and cluster deploys later in this Apache Spark with Scala course.  

Let's configure an Apache Spark cluster running on two instances of Amazon EC2.

Review key takeaways from this section on Spark running in a cluster and deploying a Scala based Spark program to the cluster.

Convenient link to download all source code used in this section

Spark SQL background, key concepts and high-level examples of CSV, JSON and mySQL (JDBC) data sources. This lecture lays the groundwork for next lectures in this course section. It provides overview examples and common patterns of Spark SQL from a Scala perspective.

Spark SQL uses a type of Resilient Distributed Dataset called DataFrames which are composed of Row objects accompanied with a schema. The schema describes the data types of each column. A DataFrame may be considered similar to a table in a traditional relational database.

Methodology

We’re going to use the baby names dataset and the spark-csv package available from Spark Packages to make our lives easier. The spark-csv package is described as a “library for parsing and querying CSV data with Apache Spark, for Spark SQL and DataFrames” This library is compatible with Spark 1.3 and above.

Let's load a JSON input source to Spark SQL’s SQLContext. This Spark SQL JSON with Scala portion of the course has two parts. The first part shows examples of JSON input sources with a specific structure. The second part warns you of something you might not expect when using Spark SQL with JSON data source.

Methodology

We are going to use two JSON inputs. We’ll start with a simple, trivial example and then move to an analysis of more realistic JSON example.

Now that we have Spark SQL experience with CSV and JSON, connecting and using a mySQL database will be easy. So, let’s cover how to use Spark SQL with Scala and a mySQL database input data source.

We’re going to load data into a database. Then, we’re going to fire up spark-shell with a command line argument to specifiy the JDBC driver needed to connect to the JDBC data source. We’ll make sure we can authenticate and then start running some queries.

Earlier in the course, we performed a simple deploy to an Apache Spark Cluster.  Let's build upon the simple example and deploy our Spark SQL code examples.    

Deploying the Spark SQL examples introduces a new challenge.  How do we deploy when our application uses 3rd party libraries such as CSV parsing and JDBC drivers?

Links to download Spark SQL code examples and videos on setting up mySQL

Spark Streaming introduction, key concepts and our approach for learning Apache Spark Streaming through examples and building our own application streaming.

Present an overview of the lessons contained in this Spark Streaming section.  For some of you, you may be able to skip the first two examples and move to a more complex Spark Streaming custom application.  

To ensure your environment is ready for more complex Spark Streaming examples, let's run through a trivial example.  This is a word count example which streams for the netcat utility found on Linux and Mac.  For windows users, check https://nmap.org/ncat/ which may be used to run this example.

Let's continue to take one step at a time as we are learning Spark Streaming.  In this example, we will build and deploy a spark streaming application to a Spark cluster.

This video demonstrates our custom Spark Streaming application and how you can configure Slack to stream your own channel content.  

I think it's important to show you running example of Spark Streaming application a 

Spark Streaming example code review.  Answers the questions -- how do I write my own custom receiver and how did the Slack Spark Streaming example work?

Our Spark Streaming with Slack program contains 3rd party libraries.  As we've seen previously in the course, we can use the sbt-assembly plugin to make "fat jars" for Spark Driver programs using 3rd party libraries.

But, what happens when things do not deploy according to plan?  

In this video, we'll cover three advanced issues when deploying to a Spark Cluster and how to address.  

1) What happens if your Spark Driver program is compiled to Scala 2.11, but you are deploying to Spark compiled to Scala 2.10?

2) What happens if your 3rd party library conflicts with your Spark Cluster? 

3) What to do if your Spark Cluster uses a jar which is older and incompatible with a jar needed by your driver program?

In this video, we'll cover three advanced issues when deploying to a Spark Cluster and how to address.  

1) What happens if your Spark Driver program is compiled to Scala 2.11, but you are deploying to Spark compiled to Scala 2.10?

2) What happens if your 3rd party library conflicts with your Spark Cluster? 

3) What to do if your Spark Cluster uses a jar which is older and incompatible with a jar needed by your driver program?

A list of resources used in this Spark Streaming section of the Apache Spark course tutorials 

Machine Learning is an exciting and growing topic of interest these days.  Let's start this section on Spark MLlib with a background on Machine Learning.  

Afterwards, we'll have a foundation of machine learning concepts when we run demos and review source code in later videos in this Spark MLlib section.

In this video, let's run a demo of a custom Spark MLlib based program so we have some context when reviewing the source code later in the course.

In this demo, we'll train our machine learning model.  Then, we'll use the trained model to make predictions on an incoming data stream.  

That's right, we're going to make machine learning based predictions on data arriving from a Spark Streaming source.  

Should be fun :)

Review of Spark MLlib based source code from the demo of the near real-time machine learning prediction model.  The model used a Spark Streaming data source which will also be analyzed.  

The code has tons of comments in it to help.  Also, the source code is available for students to download from the course repository.

Up to now, we've seen a machine learning demo of near real-time prediction of stream data and we've reviewed the custom demo code.   

So, now let's cover aspects of machine learning specific to Spark MLlib.

A suggested list of free resources for machine learning and Spark MLlib.

Conclusion of version 2 of the Apache Spark with Scala course.  We review the content of version 2 of this course, suggested next steps and ask for ideas for version 3 of the Apache Spark with Scala course.

Version 1 major release:  End of January 2015

- Spark Core and Clustering

Version 1.1, 1.2, 1.3 minor releases: February, March 2016

- Section introductions

- Add more resources to each section

- Spark SQL section

Version 2 major release: May 2016

- Spark Streaming

- Spark machine learning with Spark MLlib

Bonus lecture with access to free Spark learning resources, course coupons, tutorials and free software development, data engineering and data science books.