Learning Path: SMACK: Getting Started with the SMACK Stack

This video will give an overview of entire course

We need a data set to practice the skills learned in this course. We download the Houses Prices dataset from Kaggle for this.

Spark Notebook is a convenient environment for data analysis and reproducible research. We need to install it.

Before proceeding to load the data, we need to understand how Spark represents and handles it. This theoretical part covers it.

Now that we know the theory, we need to actually see how to load the example dataset in Spark.

Before building a statistical model of a dataset, one must have some understanding of that dataset. This video provides tools to build a visual intuition about the data in the dataset.

Another way to draw insights from the data is to look at its statistical metrics. This video describes how to compute them with Spark.

Preprocess the data before feeding it to a ML algorithm. This video describes how to do that with standard SQL/Collections methods.

SparkSQL operations are powerful, but SparkML supports some common ML operations out of the box. Learning them may greatly reduce the work to be done.

A particular kind of operation on data that is commonly used is slicing the features (taking a subset of them) based on a predicate.

Before proceeding to concrete examples of using SparkML, we need to understand its structure.

The result of data analysis is usually a model of the data in question. This video explains how to do data modeling with the ML algorithms that Spark has.

This video gives an overview of the entire course.

To find an efficient solution, we need to learn about the data processing challenges first.

It is important to know the process or pipeline of SMACK to use it better.

To use each technology, you need to understand each technology.

Now learn about data expert profiles and how data processing can be a data center operation.

We need to understand Scala hierarchy and the selection of a Scala to work with Scala. This video will teach you that.

Iterators are an important part of Scala. This video uses iterators and shows their importance.

This video shows a host of functions with Scala that includes filtering, merging, sorting and also sets, arrays queues, and stacks.

This video shows the comparison between the Actor Model and traditional OOP, then describing about the actor system and reference.

Here, we will be learning about the functioning of actors using various katas.

Apache Spark cluster-based installations can become a complex task, when we integrate Mesos, Kafka, and Cassandra from: databases, telecommunications, operating systems, and infrastructure.

Spark has four design goals: make in memory (Hadoop is not in-memory) data storage, distribute in a cluster, be fault tolerant, and be fast and efficient.

Apache Spark has its own built-in cluster standalone manager but you can run multiple cluster managers, including Apache Mesos, Hadoop YARN, and Amazon EC2.

Spark Streaming is the module for managing data flows. Much of Spark is built with the concept of RDD. It provides the concept of DStreams or Discretized Streams.

NoSQL is a distributed database with an emphasis on scalability, high availability, and ease of administration, the opposite of established relational databases.

The task of creating a scalable database massively decentralized, optimized for read operations, painlessly modifying data structures. The solution was found by combining two existing technologies that is Google's BigTable and Amazon's Dynamo.

Cassandra offers to create a back up on the local computer. It creates a copy of the base using a snapshot. It is possible to make a snapshot of all the key spaces. Compression increases the cluster nodes capacity, reducing the data size on the disk.

If you use an incremental backup, it is also necessary to provide the incremental backups created after the snapshot. There are multiple ways to perform a recovery from the snapshot.

Work with DBMS optimization

The Spark Cassandra connector is a client used to achieve this connection, but this client is special because it has been designed specifically for Spark and not for a specific language.

In this video, you will learn the basics of the Spark Cassandra connector

Spark streaming allows for handling and processing of high throughput and fault tolerant live data streams. In this video, you will learn about Spark Cassandra streaming and create a stream.

Once our Spark Cassandra is set up, we'll look at the different operations we can perform with Cassandra.

In this video, we will use the Akka Cassandra connector to build a simple Akka application, make HTTP requests, and store the data in Cassandra.

Increasing data requires better data processing systems. Hence, Kafka comes into picture. In this video, you will learn about the features of Kafka and basics of Kafka.

We need to install Kafka to work with it. This video will enable you to do that.

Clusters are Kafka’s Publisher-subscriber messaging systems. In this video, you will learn to program with them.

In this video, we will look at how the Kafka architecture is designed and understand the components that make it what it is.

Producers are applications that create messages and publish them to the broker. You need to understand the working of producers.

Consumers are applications that consume the messages published by the broker. So they are the next step in the Kafka architecture.

To process large volumes of data, we require to integrate Kafka with other big data tools. Integration teaches us that. Also there are numerous tools provided by Kafka to manage features. We will learn about that in administration.

In this video, we will be looking at the relation between Akka and Spark and Kafka and Akka.

In this video, we will review the connectors between Kafka and Cassandra.

In this video, you will be introduced to Mesos and learn about the Mesos architecture.

Resource allocation module of Mesos decides quantity of resources allocated to each framework. Hence, it is important to know about the resource allocation in Mesos.

If you don’t want to use cloud services from Amazon, Google, or Microsoft, we can set up our cluster on our private data center. This video will teach you how to do that.

We need frameworks to deploy, discover, balance load, and handle failure of services. In this video, we will look at the frameworks that are used for service management.

Aurora is a Mesos framework for long running services and cron jobs. Learn about job scheduling with Aurora.

Singularity is a platform that enables deploying and running services and scheduled jobs in the cloud or data centers. Combined with Apache Mesos, it provides efficient management of the underlying processes life cycle and effective use of cluster resource. Let's see what it is all about.

In this video, you will learn how to run Apache Spark on Mesos

In this video, we will deploy Apache Cassandra on Apache Mesos with the help of Marathon.

In this video, we will deploy Apache Kafka on Apache Mesos.