Learn Apache Spark and Scala from Scratch

Name: Learn Apache Spark and Scala from Scratch
Rating: 3.7 (2 reviews)

A Basic to Advanced Overview for processing Big Data with Spark

Created byAnurag Kaushik

Last updated 12/2022

English

What you'll learn

OOPS and Functional Programming in Scala
Apache Spark Framework
Advanced Spark Programming
Integrating Spark with Kafka
Spark MLib - Machine Learning
Spark Streaming, SparkSQL, Spark GraphX etc.

Course content

11 sections • 116 lectures • 9h 23m total length

Introduction to Scala4:02
Getting Started with Scala in Windows2:39
Downloading VMware and UBUNTU ISO Image1:25
Installing VMware Workstation Player and UBUNTU3:08
Installing Java in UBUNTU4:40
Downloading and Installing Spark and Scala in UBUNTU3:31
Scala Running Modes6:44
Some Keypoints about Scala2:18
Scala Type Inference and Explicit Types2:52
Scala Variables7:13
Scala Class and Objects9:21
Access Modifiers in Scala7:24
Learn how Scala uses access modifiers to control member visibility: private restricts access to the defining class, protected allows subclass access, and public remains accessible everywhere by default.
If Else Statements in Scala8:00
Operators in Scala16:35
Loops in Scala11:06

Functions and Procedures in Scala5:19
Call By Name Parameter3:05
Explore Scala's call by name parameters, where a code block is passed and evaluated each time the parameter is accessed, delaying computation versus by value parameters.
Functions with Named Arguments2:15
Functions with Variable Arguments2:32
Recursion Functions2:06
Default Parameters for a Function2:57
Nested Functions4:41
Anonymous Functions3:17
Strings in Scala7:27
Arrays in Scala9:11
Explore arrays in Scala, including declaring fixed-size arrays, indexing, multi-dimensional arrays, concatenation, and processing elements with loops and simple examples.
Scala Collections1:58
Explore scala collections, including lists, sets, maps, tuples, and options. Understand mutable vs immutable, and remember to start with immutable collections before switching to mutable.
Lists in Scala9:50
Sets in Scala8:28
Maps in Scala9:38
Tuples in Scala6:15
Options in Scala9:20
Exception Handling in Scala3:45
Pattern Matching6:36
Scala Traits4:42
Explore Scala traits as a midway point between Java interfaces and multiple inheritance, enabling mixing traits into classes, partial implementations, and guidelines for when to use traits versus abstract classes.
Scala Files Input Output6:06
Extractors in Scala5:12

Big Data Overview3:08
Facts about Big Data3:32
Explore the facts about big data, showing how social networks, mobile devices, and video platforms generate massive volumes—from tweets per minute to petabytes per year.
Big Data Scenarios2:16
Apache Hadoop Framework3:00
Top Hadoop Users2:20
Explore how top Hadoop users like Amazon Web Services, IBM Infosphere BigInsights, Cloudera, MapR, and Datastax Enterprise harness Hadoop based software to manage big data cost-effectively and enable enterprise analytics.
History of Hadoop2:04
Difference between RDBMS and Hadoop1:20
Cluster Modes in Hadoop1:07
Hadoop Ecosystem5:23
HDFS Daemons and MapReduce Daemons2:57
Hadoop Cluster Architecture3:08
Explore Hadoop cluster architecture, including client, master (name node, secondary name node, job tracker), and slave (data node, task tracker) roles for storing and processing unstructured data with MapReduce.
Top Reasons Why you should Learn Hadoop7:49
Hadoop Distributions and Compatibilities0:46
Hadoop Ecosystem in Detail18:40
Explore the Hadoop ecosystem and its core components, including HDFS, MapReduce, Yarn, Hive, Pig, HBase, Hcatalog, and more, with insights into their roles and data processing capabilities.
Hadoop Distributed File System0:46
HDFS Files and Blocks4:06
Explore how HDFS splits files into 64 MB blocks, stores them on data nodes under the NameNode, and uses threefold replication with rack awareness to ensure fault tolerance.
HDFS Components and Architecture9:21
Explore HDFS components and architecture, including the name node as master and data nodes for block storage; learn about blocks, replication, FS image, edit logs, and heartbeat coordination.
HDFS File Read and Write5:07

Batch and Real Time Analytics3:29
Why Spark when Hadoop is Already there1:49
Introduction to Apache Spark3:53
Features of Apache Spark3:12
Users and Use Cases of Apache Spark11:00
Job Execution Flow and Spark Execution0:51
Learn the job execution flow and Spark execution, contrasting Hadoop's disk-based data access with Spark's in-memory processing and noting cache steps and deployment options Mesos, Yarn, or Spark's cluster manager.
Spark Unified Stack7:32
Complete Picture of Apache Spark4:39
Apache spark Architecture8:22
Why Spark with Scala1:27
Top Companies Using Spark4:04

Requirements

Intermediate programming experience in Python or Scala
Beginner experience with the DataFrame API
Basic understanding of Machine Learning concepts

Description

This course on Apache Spark and Scala aims at providing an advanced expertise in big data Hadoop ecosystem. This course will provide a standard skillset which helps one become a specialist on the top of Big data Hadoop developer.

Apache Spark is a lightning-fast cluster computing designed for fast computation.

The course starts with a detailed description on limitations of mapreduce and how Spark can help overcome them. Further it covers a deeper dive into the Scala programming language.

Moving on it covers Spark as a standalone cluster and an understanding of Resiliient Distributed Datasets.

The course also covers concepts of Spark SQL using SQL queries through SQL context and Hive Queries through Hive context.

This course certainly provides material required for building a career path from Big data Hadoop developer to BIg data Hadoop architect.

This course has been prepared for professionals aspiring to learn the basics of Big Data Analytics using Spark Framework and become a Spark Developer. In addition, it would be useful for Analytics Professionals and ETL developers as well.

Before you start proceeding with this course, we assume that you have prior exposure to Scala programming, database concepts, and any of the Linux operating system flavors.

Who this course is for:

Professionals aspiring to learn the basics of Big Data Analytics
Spark Developer
Analytics Professionals
ETL Developers

Learn Apache Spark and Scala from Scratch

What you'll learn

Explore related topics

Course content

Module115 lectures • 1hr 31min

Module 221 lectures • 1hr 55min

Module 318 lectures • 1hr 17min

Module 411 lectures • 50min

Module 510 lectures • 1hr 1min

Module 64 lectures • 26min

Module 77 lectures • 33min

Module 811 lectures • 51min

Module 97 lectures • 17min

Module 107 lectures • 21min

Requirements

Description

Who this course is for: