Databricks - Master Azure Databricks for Data Engineers

Name: Databricks - Master Azure Databricks for Data Engineers
Rating: 4.6 (3831 reviews)

Learn Azure Databricks for professional data engineers using PySpark and Spark SQL with an end-to-end capstone project

Created byLearning Journal, Prashant Kumar Pandey

Last updated 5/2026

English

What you'll learn

Databricks in Azure Cloud
Working with DBFS and Mounting Storage
Unity Catalog - Configuring and Working
Unity Catalog User Provisioning and Security
Working with Delta Lake and Delta Tables
Manual and Automatic Schema Evolution
Incremental Ingestion into Lakehouse
Databricks Autoloader
Delta Live Tables and DLT Pipelines
Databricks Repos and Databricks Workflow
Databricks Rest API and CLI
Capstone Project

Course content

12 sections • 91 lectures • 17h 32m total length

Course Prerequisites2:14
Identify prerequisite topics such as Spark SQL, Spark DataFrame API, Spark Structured Streaming API, Python basics, and Spark architecture and internals to benefit from the course.
About the Course5:07
How to access Course Material and Resources11:51
Learn how to access and download course resources, including notebooks, sample data, and capstone project, then import notebooks into Azure Databricks and upload data to cloud storage.
Note for Students - Before Start2:05
Encourage students to share honest reviews and five-star ratings to support ongoing course updates and high-quality content, with a 30-day refund if the course doesn't meet expectations.

What will you learn in this section1:51
Introduction to Delta Lake6:25
Discover delta lake, an open source storage framework between processing engine and cloud storage, enabling acid transactions, delete/update/merge, schema enforcement, data versioning with time travel, and streaming and batch unification.
Creating Delta Table15:27
Sharing data for External Delta Table12:48
Reading Delta Table10:04
Delta Table Operations21:58
Master delta table operations in Spark, including delete, update, and merge, with Spark SQL and Delta table API in Python.
Delta Table Time Travel20:56
Convert Parquet to Delta8:22
Convert a partitioned parquet data set to a delta data set in place with the convert to delta command, enabling in-place migration and delta log creation.
Delta Table Schema Validation21:12
Delta Table Schema Evolution28:56
Look Inside Delta Table18:28
Delta Table Utilities and Optimization43:41

What will you learn in this section1:14
Explore incremental data ingestion in lakehouses, covering architecture and use cases, then learn copy command, spark streaming, and auto loader for ingestion with manual and automatic schema evolution.
Architecture and Need for Incremental Ingestion6:05
Using Copy Into with Manual Schema Evolution17:53
Learn to use copy into to ingest landing zone data into a bronze table with a fixed schema, and apply manual schema evolution to handle new columns.
Using Copy Into with Automatic Schema Evolution13:41
Master Databricks copy into with automatic schema evolution to ingest csv data from a landing zone into a schema-less delta table, inferring and merging schema on the fly.
Streaming Ingestion with Manual Schema Evolution9:43
Streaming Ingestion with Automatic Schema Evolution8:55
Introduction to Databricks Autoloader5:10
Explore Databricks Auto Loader, a cloud-native, spark streaming framework that efficiently ingests new files from cloud storage with incremental listing, optimized reads, and optional data landing notifications.
Autoloader with Automatic Schema Evolution31:54

What will you learn in this section1:31
Introduction to Databricks DLT6:21
Understand DLT Use Case Scenario10:33
Build delta live tables pipelines from landing zone to bronze, silver, gold layers with incremental processing, apply QCD type two, implement CDC with merge for UK 2022 daily sales report.
Setup DLT Scenario Dataset6:06
Creating DLT Workload in SQL45:44
Creating DLT Pipeline for your Workload19:41
Learn to create and schedule a delta live table pipeline using the UI, connect your code from workspace or repo, and run it against Unity Catalog or Hive metastore.
Creating DLT Workload in Python46:57
Learn to build delta live tables pipelines in python, creating bronze raw tables, cleaning with data quality, and silver scd type 2 merges, plus daily materialized views for final analytics.

Requirements

Python Programming Language
Apache Spark and Dataframe APIs using Python
Spark Structured Streaming APIs using Python

Description

About the Course

I am creating Databricks - Master Azure Databricks for Data Engineers using the Azure cloud platform. This course will help you learn the following things.

Databricks in Azure Cloud
Working with DBFS and Mounting Storage
Unity Catalog - Configuring and Working
Unity Catalog User Provisioning and Security
Working with Delta Lake and Delta Tables
Manual and Automatic Schema Evolution
Incremental Ingestion into Lakehouse
Databricks Autoloader
Delta Live Tables and DLT Pipelines
Databricks Repos and Databricks Workflow
Databricks Rest API and CLI

Capstone Project

This course also includes an End-To-End Capstone project. The project will help you understand the real-life project design, coding, implementation, testing, and CI/CD approach.

Who should take this Course?

I designed this course for data engineers who are willing to develop Lakehouse projects following the Medallion architecture approach using the Databrick cloud platform. I am also creating this course for data and solution architects responsible for designing and building the organization’s Lakehouse platform infrastructure. Another group of people is the managers and architects who do not directly work with Lakehouse implementation. Still, they work with those implementing Lakehouse at the ground level.

Spark Version used in the Course.

This course uses Databricks in Azure Cloud and Apache Spark 3.5. I have tested all the source codes and examples used in this course on Azure Databricks Cloud using Databricks Runtime 13.3.

Who this course is for:

Data Engineers
Data Engineering Solution Architects

Databricks - Master Azure Databricks for Data Engineers

What you'll learn

Explore related topics

Course content

Before you start4 lectures • 21min

Introduction3 lectures • 34min

Getting Started6 lectures • 1hr 1min

Working in Databricks Workspace5 lectures • 57min

Working with Databricks File System - DBFS4 lectures • 56min

Working with Unity Catalog5 lectures • 1hr 37min

Working with Delta Lake and Delta Tables12 lectures • 3hr 30min

Working with Databricks Incremental Ingestion Tools8 lectures • 1hr 35min

Working with Databricks Delta Live Tables (DLT)7 lectures • 2hr 17min

Databricks Project and Automation Features5 lectures • 1hr 9min

Requirements

Description

Who this course is for: