Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA Amazon AWS CompTIA Security+ Microsoft AZ-900
Photoshop Graphic Design Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Mindfulness Personal Development Personal Transformation Meditation Life Purpose Coaching Neuroscience
Web Development JavaScript React CSS Angular PHP WordPress Node.Js Python
Google Flutter Android Development iOS Development Swift React Native Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Retargeting
SQL Microsoft Power BI Tableau Business Analysis Business Intelligence MySQL Data Analysis Data Modeling Big Data
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Online Business Business Plan Startup Freelancing Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee

This course includes:

  • 12 hours on-demand video
  • 5 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
Development Data Science Apache Spark

Databricks Fundamentals & Apache Spark Core

Learn how to process big-data using Databricks & Apache Spark 2.4 and 3.0.0 - DataFrame API and Spark SQL
Bestseller
Rating: 4.4 out of 54.4 (203 ratings)
9,600 students
Created by Wadson Guimatsa
Last updated 7/2020
English
30-Day Money-Back Guarantee

What you'll learn

  • Databricks
  • Apache Spark Architecture
  • Apache Spark DataFrame API
  • Apache Spark SQL
  • Selecting, and manipulating columns of a DataFrame
  • Filtering, dropping, sorting rows of a DataFrame
  • Joining, reading, writing and partitioning DataFrames
  • Aggregating DataFrames rows
  • Working with User Defined Functions
  • Use the DataFrameWriter API

Course content

7 sections • 71 lectures • 12h 8m total length

  • Preview02:20
  • Preview02:26
  • Preview04:30
  • Overview of the dataset
    05:46
  • Install the notebooks
    02:20

  • Introduction to databricks
    08:47
  • Write your first Apache Spark Code
    09:59
  • Preview08:37
  • Practice: Find customer with the same birthday as you
    02:08

  • Create a DataFrame from a CSV file
    06:10
  • Configure options to read a CSV file
    06:20
  • Preview05:19
  • How to reference columns of a DataFrame
    05:20
  • Understand the DataFrame Schema: Part 1
    02:22
  • Understand the DataFrame Schema: Part 2
    04:11
  • Specify a DataFrame Schema using a DDL-formatted string : Part 1
    03:20
  • Specify a DataFrame Schema using a DDL-formatted string : Part 2
    05:37
  • Spark Architecture: The Organization of a DataFrame
    02:57

  • Adding columns to a DataFrame
    09:44
  • Renaming columns of a DataFrame
    04:43
  • Removing columns from a DataFrame
    01:27
  • Filtering rows from a DataFrame
    12:09
  • Joining multiple DataFrames: Part 1
    02:44
  • Joining multiple DataFrames: Part 2
    08:45
  • Aggregation: Count
    04:37
  • Aggregation: Count Distinct
    02:08
  • Aggregation: Get the Min value
    09:54
  • Aggregation: Get the Max value
    02:06
  • Aggregation: Get the Sum and SumDistinct
    05:59
  • Aggregation: Average and Mean
    05:37
  • Aggregation: Grouping data - Part 1
    01:09
  • Aggregation: Grouping data - Part 2
    06:45
  • Practice: Business Query 1
    22:06
  • Practice: Business Query 2
    17:37
  • Apache Spark Architecture: How Apache Spark Transforms data Internally
    31:42
  • User Defined Function
    17:22

  • Run SQL on a DataFrame: TempView
    11:17
  • Run SQL on a DataFrame: GlobalView
    06:18
  • Databases: List, Create, Delete, Select
    09:07
  • Tables: Unmanaged
    10:14
  • Tables: Managed
    14:05
  • SQL Fundamentals: Select Clause & Select Expression
    18:13
  • SQL Fundamentals: Where Clause, Equality Checks
    12:12
  • SQL Fundamentals: Handling NULLs in Where Clause
    05:09
  • SQL Fundamentals: Aggregations - Sum, Count, AVG, Mean
    14:29
  • SQL Fundamentals: Group By Clause
    11:57
  • SQL Fundamentals: Having Clause
    13:47
  • SQL Fundamentals: Order By Clause
    04:56
  • SQL Fundamentals: Inner Joins
    10:10
  • SQL Fundamentals: Left Outer Joins
    10:06
  • SQL Fundamentals: Right Outer Joins
    07:12
  • SQL Fundamentals: Predicates and Operators, like predicate
    05:56
  • SQL Fundamentals: Case Expressions
    04:50
  • Practice : Business Query 3
    18:15
  • Practice: Business Query 4
    18:14
  • Practice: Business Query 5
    07:02

  • Preview09:48
  • Converting literals to Spark Types: The lit function
    05:51
  • Working with booleans
    18:43
  • Working with numbers
    19:22
  • Working with strings
    22:56
  • Working with dates and timestamps
    26:25
  • Complex Types: Structs
    16:38
  • Complex Types: Arrays
    17:21
  • Complex Types: Maps
    11:54
  • Handling NULL Values: Drop NULL Values
    14:57
  • Handling NULL Values: Replace NULL Values
    09:33

  • DataFrameReader: Read CSV Files
    32:23
  • Preview16:34
  • DataFrameWriter: Write Data
    30:14
  • Create DataFrame manually
    07:05

Requirements

  • Basic Scala knowledge
  • Basic SQL knowledge

Description

Welcome to this course on Databricks and Apache Spark 2.4 and 3.0.0

Apache Spark is a Big Data Processing Framework that runs at scale.
In this course, we will learn how to write Spark Applications using Scala and SQL.

Databricks is a company founded by the creator of Apache Spark.
Databricks offers a managed and optimized version of Apache Spark that runs in the cloud.

The main focus of this course is to teach you how to use the DataFrame API & SQL to accomplish tasks such as:

  • Write and run Apache Spark code using Databricks

  • Read and Write Data from the Databricks File System - DBFS

  • Explain how Apache Spark runs on a cluster with multiple Nodes

Use the DataFrame API and SQL to perform data manipulation tasks such as

  • Selecting, renaming and manipulating columns

  • Filtering, dropping and aggregating rows

  • Joining DataFrames

  • Create UDFs and use them with DataFrame API or Spark SQL

  • Writing DataFrames to external storage systems

List and explain the element of Apache Spark execution hierarchy such as

  • Jobs

  • Stages

  • Tasks


Who this course is for:

  • Software developers curious about big-data, data engeneering and data science
  • Beginner data engineer who want to learn how to do work with databricks
  • Beginner data scientist who want to learn how to do work with databricks

Instructor

Wadson Guimatsa
Data Engineer
Wadson Guimatsa
  • 4.5 Instructor Rating
  • 280 Reviews
  • 9,969 Students
  • 2 Courses

I'm a software developer specialized in building data-intensive applications.

I've been developing software for over 10 years.

I've worked for Industries that are very data-intensive such as the financials and industrial image processing.

Over the years, the volume of data produced by systems and humans outgrew the storage and compute capacity of the legacy RDBMS systems, and therefore I had to learn how to use the new tools and frameworks to process Big-Data

As a data engineer, I'm very motivated and passionate about building applications that can leverage the power and flexibility of cloud computing and big-data processing frameworks.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.