Apache Spark with Databricks

Name: Apache Spark with Databricks
Rating: 3.4 (301 reviews)

Course to implement Big Data's Apache Spark on Databricks using a Microsoft's cloud service - Azure

Created byBig Data Trunk

Last updated 9/2019

English

English [Auto],

What you'll learn

This course will provide you an in depth knowledge of apache Spark and how to work with spark using Azure Databricks.
You will learn to Provision your own Databricks workspace using Azure cloud.
You will be able to create application on Azure Databricks after completing the course
You will be able to process continual streams of data with Spark streaming using azure event Hub
Transform structured data using SparkSQL and Dataframes
Build a binary classification application using the Mlib pipelines API.

Course content

7 sections • 34 lectures • 3h 0m total length

Course Overview2:30
An overview of our course
Big Data trunk Introduction !!2:34
Something about us … the BIG DATA TRUNK

Big Data before Spark11:42
How we use to work with big Data before the introduction of Spark
Big Data After Spark7:30
How the Big data world drastically changed after the introduction of Spark
Introduction to Apache Spark4:11
An Introduction to APACHE SPARK
Spark - RDD6:01
Introduction to Spark RDD
Lab - Ways for creating RDD6:32
Create rdds in Spark using Spark context and the parallelize method, first with numeric data then from a text file, and verify contents by printing results.
Quiz: Spark Introduction
Spark Application Architecture4:52
Brief Discussion about Spark Architecture
Spark Operations - Transformation & Action7:18
Types of Operations done on Spark RDD. We will mainly discuss about Transformation and actions. We will also discuss few sample codes to illustrate in a better way.
Lab - Transformation & Action7:42
DAG (Direct Acyclic Graph)9:20
Quiz-2

Spark Databricks Analytics in Azure6:36
An introduction to Azure Databricks
Why Databricks ?5:24
Here in video we will briefly discuss why we need Azure Databricks even if we have Apache Spark
Provisioning Databricks workspace in Azure portal3:15
We will provision Azure Databricks cluster and also we will learn to create a cluster in Azure Databricks
Using Databricks community Edition4:34
Learn how to sign up for the Databricks community edition, create a free cluster with a private node, and run notebooks in Python or Scala.
Quiz-3

Spark SQL8:01
We will use Jupiter notebook to run queries on set of external data. Also we will learn to use SQL & Panda queries in notebook.
After that we will learn to plot different types of charts
Quiz-4
Developing Spark application10:28
We will Develop a word count program. Also we will run this program on set of external data by removing special characters from the dataset.
Removing Stopwords2:00
Developing Spark ML application14:06
In this video we will cover Databricks runtime Machine learning. We will use Binay Composition and decision tree model to train our Model
Developing Spark Streaming applications (real time Twitter Data)7:47
We will work on Streaming data from the twitter. We will use Azure eventHub for reading the Twits from Twitter.
Optimizing Spark Performance10:06
How the Spark can be optimized to perform more efficiently and effectively.
Quiz-5

Spark Vs Hadoop / mapReduce3:50
Compare Spark and Hadoop MapReduce, highlighting in-memory processing, batch versus streaming data, and Spark Streaming for near real-time processing.
Spark Core1:59
Something more about RDD5:43
Type Safety in Dataset3:16
Justify Lazy Evaluation in Spark2:15
Spark uses lazy evaluation for transformations, building a DAG that delays execution until an action triggers it. This reduces driver-to-cluster trips, lowers overhead, and speeds up processing.
Explain Fault Tolerance in Spark5:44
Explore fault tolerance in spark streaming via lineage-based recomputation, data replication across executors, reliable versus unreliable receivers, and write-ahead logs for durability.
Coalesce and Repartition2:57
Spark Executor Memory1:06
Explore spark executor memory, the heap allocation for each spark application, and how memory is managed per worker, controlled by the executor memory property across the cluster.
Garbage Collection Optimization in Spark6:33
Learn More on Spark Interview0:06

Requirements

Some Prior scripting knowledge, anyways we will be explainig all the codes line by line whichever we will be using in our labs
Free or paid subscription for Microsoft Azure portal.

Description

In this course you will learn the basics of creating Spark jobs, loading data, and working with data. You’ll also get an introduction to running machine learning algorithms and working with streaming data. Databricks lets you start writing Spark queries instantly so you can focus on your data problems.

Azure Databricks accelerate big data analytics and artificial intelligence (AI) solutions, a fast, easy and collaborative Apache Spark–based analytics service.

Why Azure Databricks?

Productive : Launch your new Apache Spark environment in minutes.

Scalable : Globally scale your analytics and machine learning projects.

Trusted : Help protect your data and business with Azure AD integration, role-based controls and enterprise-grade SLAs.

Flexible : Build machine learning and AI solutions with your choice of language and deep learning frameworks.

We believe that when you learn something, you should be able to apply it somewhere. So, in this course, we are also providing you with some of the important spark interview questions , which will help you to crack the interview with flying colors.

Who this course is for:

Anyone who wants to learn Spark using Azure Databricks

Apache Spark with Databricks

What you'll learn

Explore related topics

Course content

Overview2 lectures • 5min

Apache Spark Introduction9 lectures • 1hr 5min

Databricks with Microsoft Azure4 lectures • 20min

Understanding Cluster and Notebooks in Databricks1 lecture • 3min

Working with Spark in Databricks6 lectures • 52min

Spark Interview10 lectures • 33min

Bonus Section2 lectures • 1min

Requirements

Description

Who this course is for: