Streaming Big Data with Spark Streaming & Scala - Hands On!

We start our crash course in the Scala programming language by covering some basics of the language: types and variables, printing, and boolean comparisons.

Part 2 of our introduction to the basics of Scala programming, and a simple exercise to get you writing your own Scala code.

Our Scala crash course continues, illustrating various means of flow control in Scala. For loops, do/while loops, while loops, etc.

Scala is a functional programming language, and so understanding how functions work and are treated in Scala is hugely important! This lecture covers the fundamentals, and lets you put it into practice.

We wrap up our Scala crash course with commonly used data structures using in Spark with Scala. Tuples, lists, and maps.

Before you can learn about Spark Streaming, you need to understand how Spark itself works at a high level! This covers the why & how of Apache Spark, of which Spark Streaming is a component.

The fundamental object of Spark programming is the Resilient Distributed Dataset (RDD), and this is used not just in Spark but also within Spark Streaming scripts. This lecture explains what they are, and what you can do with them.

Let's walk through and actually run a simple Spark script that counts the number of occurrences of each word in a book.

We finally have all the pre-requisite knowledge to start talking about Spark Streaming itself in more detail! We'll cover how it works, what it's for, and its architecture.

Now that we know more, let's go revisit that first Spark Streaming application we ran in lecture 2, and dive into how it really works.

Windowing allows you to analyze streaming data over a sliding window of time, which lets you do much more than just transform streaming data and store it someplace else. We'll cover the concepts of the batch, window, and slide intervals, and how they work together to let you aggregate streaming data over some period of time.

How can Spark Streaming do so much work continuously in a reliable manner? We'll uncover some of its tricks for reliability, as well as tips for configuring Spark Streaming to be as reliable as possible.

We'll build on our "print tweets" example to actually store the incoming Tweets to disk, and illustrate how Spark Streaming can handle file output.

Compute the average length of a Tweet, using windowing in Spark Streaming.

This example will listen to an Apache access log stream, and raise an alarm if too many errors are returned by the server in real time.

We'll integrate Spark Streaming with Spark SQL, allowing us to run SQL queries on data as it is streamed in! Again we will use Apache logs as an example.

Spark 2.0 introduced experimental support for Structured Streaming, a new DataSet-based API for Spark Streaming that is bound to become increasingly important. Learn how it works.

As an example, we'll stream Apache access logs in from a directory, and use Structured Streaming to count up status codes over a one-hour moving window.

Flume is a popular technology for publishing log information at large scale, especially on a Hadoop cluster. We'll illustrate how to set up both push-based and pull-based Flume configurations with Spark Streaming, and discuss the tradeoffs of each.

Amazon's Kinesis Streaming service is basically Kafka on AWS. If you're working with an AWS/EC2 cluster, you'll want to know how to integrate Spark Streaming with Kinesis - and that's what this lecture covers.

What if you need to integrate Spark Streaming with some proprietary system that does not have an existing connection library? Well, you can always write your own Receiver class. This example shows you how and actually lets you build and run one.

Cassandra is a popular "NoSQL" database that can be used to provide fast access to massive data sets to real-time applications. Dumping data transformed by Spark Streaming into a Cassandra database can expose that data you your larger, real-time services. We'll show you how and actually run a simple example.

Spark has the ability to track arbitrary state across streams of data as they come in, such as web sessions, running totals, etc. This example shows you how it all works, and challenges you to track your own state using our example as a baseline.

Spark Streaming integrates with some of Spark's MLLib (Machine Learning Library) capabilities. This example creates a real-time K-Means clustering example; it does unsupervised machine learning that continually gets better as more training data feeds into it.

Spark Streaming can also feed data in real-time to linear regression models, that get better over time as more data is fed into them. This example shows linear regression in action with Spark Streaming.

Your production applications won't be run from within the Scala IDE; you'll need to run them from a command line, and potentially on a cluster. The spark-submit command is used for this. We'll show you how to package up your application and run it using spark-submit from a command prompt.

If your Spark Streaming application has external library dependencies that might not be already present on every machine in your cluster, the SBT tool can manage those dependencies for you, and package them into the JAR file you run with spark-submit. We'll show you how it works with a real example.

We'll run our simple word count example on a real cluster, using Amazon's Elastic MapReduce service! This just shows you what's involved in running a Spark Streaming job on a real cluster as opposed to your desktop; there are a few parameters to spark-submit you need to worry about, and getting your scripts and data in the right place is also something you need to deal with.

Spark jobs rarely run perfectly, if at all, on the first try - some tuning and debugging is usually required, and arriving at the right scale of your cluster is also necessary. We'll cover some performance tips, and how to troubleshoot what's going on with a Spark Streaming job running on a cluster.

Want to learn more about Spark Streaming? Here are a few books and other resources I've found valuable.