Databricks Fundamentals & Apache Spark Core

Name: Databricks Fundamentals & Apache Spark Core
Rating: 4.5 (656 reviews)

Learn how to process big-data using Databricks & Apache Spark 2.4 and 3.0.0 - DataFrame API and Spark SQL

Bestseller

(656 ratings)

12,732 students

Created by Wadson Guimatsa

Last updated 9/2021

English

English [Auto]

30-Day Money-Back Guarantee

What you'll learn

Databricks
Apache Spark Architecture
Apache Spark DataFrame API
Apache Spark SQL
Selecting, and manipulating columns of a DataFrame
Filtering, dropping, sorting rows of a DataFrame
Joining, reading, writing and partitioning DataFrames
Aggregating DataFrames rows
Working with User Defined Functions
Use the DataFrameWriter API

Curated for the Udemy Business collection

7 sections • 71 lectures • 12h 8m total length

Preview02:20
Preview02:26
Preview04:30
Overview of the dataset
05:46
Install the notebooks
02:20

Create a DataFrame from a CSV file
06:10
Configure options to read a CSV file
06:20
Preview05:19
How to reference columns of a DataFrame
05:20
Understand the DataFrame Schema: Part 1
02:22
Understand the DataFrame Schema: Part 2
04:11
Specify a DataFrame Schema using a DDL-formatted string : Part 1
03:20
Specify a DataFrame Schema using a DDL-formatted string : Part 2
05:37
Spark Architecture: The Organization of a DataFrame
02:57

Adding columns to a DataFrame
09:44
Renaming columns of a DataFrame
04:43
Removing columns from a DataFrame
01:27
Filtering rows from a DataFrame
12:09
Joining multiple DataFrames: Part 1
02:44
Joining multiple DataFrames: Part 2
08:45
Aggregation: Count
04:37
Aggregation: Count Distinct
02:08
Aggregation: Get the Min value
09:54
Aggregation: Get the Max value
02:06
Aggregation: Get the Sum and SumDistinct
05:59
Aggregation: Average and Mean
05:37
Aggregation: Grouping data - Part 1
01:09
Aggregation: Grouping data - Part 2
06:45
Practice: Business Query 1
22:06
Practice: Business Query 2
17:37
Apache Spark Architecture: How Apache Spark Transforms data Internally
31:42
User Defined Function
17:22

Run SQL on a DataFrame: TempView
11:17
Run SQL on a DataFrame: GlobalView
06:18
Databases: List, Create, Delete, Select
09:07
Tables: Unmanaged
10:14
Tables: Managed
14:05
SQL Fundamentals: Select Clause & Select Expression
18:13
SQL Fundamentals: Where Clause, Equality Checks
12:12
SQL Fundamentals: Handling NULLs in Where Clause
05:09
SQL Fundamentals: Aggregations - Sum, Count, AVG, Mean
14:29
SQL Fundamentals: Group By Clause
11:57
SQL Fundamentals: Having Clause
13:47
SQL Fundamentals: Order By Clause
04:56
SQL Fundamentals: Inner Joins
10:10
SQL Fundamentals: Left Outer Joins
10:06
SQL Fundamentals: Right Outer Joins
07:12
SQL Fundamentals: Predicates and Operators, like predicate
05:56
SQL Fundamentals: Case Expressions
04:50
Practice : Business Query 3
18:15
Practice: Business Query 4
18:14
Practice: Business Query 5
07:02

Requirements

Basic Scala knowledge
Basic SQL knowledge

Description

Welcome to this course on Databricks and Apache Spark 2.4 and 3.0.0

Apache Spark is a Big Data Processing Framework that runs at scale.
In this course, we will learn how to write Spark Applications using Scala and SQL.

Databricks is a company founded by the creator of Apache Spark.
Databricks offers a managed and optimized version of Apache Spark that runs in the cloud.

The main focus of this course is to teach you how to use the DataFrame API & SQL to accomplish tasks such as:

Write and run Apache Spark code using Databricks

Read and Write Data from the Databricks File System - DBFS

Explain how Apache Spark runs on a cluster with multiple Nodes

Use the DataFrame API and SQL to perform data manipulation tasks such as

Selecting, renaming and manipulating columns

Filtering, dropping and aggregating rows

Joining DataFrames

Create UDFs and use them with DataFrame API or Spark SQL

Writing DataFrames to external storage systems

List and explain the element of Apache Spark execution hierarchy such as

Jobs

Stages

Tasks

Who this course is for:

Software developers curious about big-data, data engeneering and data science
Beginner data engineer who want to learn how to do work with databricks
Beginner data scientist who want to learn how to do work with databricks

Wadson Guimatsa

Data Engineer

4.5 Instructor Rating
1,281 Reviews
15,701 Students
2 Courses

I'm a software developer specialized in building data-intensive applications.

I've been developing software for over 10 years.

I've worked for Industries that are very data-intensive such as the financials and industrial image processing.

Over the years, the volume of data produced by systems and humans outgrew the storage and compute capacity of the legacy RDBMS systems, and therefore I had to learn how to use the new tools and frameworks to process Big-Data

As a data engineer, I'm very motivated and passionate about building applications that can leverage the power and flexibility of cloud computing and big-data processing frameworks.

Databricks Fundamentals & Apache Spark Core

What you'll learn

Course content

Setup5 lectures • 17min

Introduction to Databricks and Apache Spark4 lectures • 30min

The DataFrame API: Basics9 lectures • 42min

The DataFrame API: Transforming Data18 lectures • 2hr 47min

Spark SQL & SQL Fundamentals20 lectures • 3hr 33min

Working with different type of data11 lectures • 2hr 53min

Data Sources4 lectures • 1hr 26min

Requirements

Description

Who this course is for:

Instructor