Databricks - Master Azure Databricks for Data Engineers

Learn how data engineers ingest, transform, and model data using the lake house medallion architecture (bronze, silver, gold) on Databricks and Apache Spark, with batch, streaming, and near real-time options.

Apache Spark powers distributed data engineering, stream processing, and machine learning with Spark SQL, data frame APIs, and structured streaming on a cluster, while Databricks extends it for enterprise use.

Explore the Databricks platform, a cloud-native spark solution for lakehouse architectures with Delta Lake, Unity Catalog, Photon Query Engine, and Medallion Architecture, notebooks, and governance across Azure, AWS, and GCP.

Learn how to set up an azure cloud account, navigate the azure portal and databricks workspace UI, create your first azure databricks workspace, and understand azure databricks architecture.

Create your Azure free account to access 200 USD credit for 30 days, plus 12 months of free services and 40+ always free services.

Sign in to the Azure portal and explore its web-based interface to manage subscriptions, resource groups, dashboards, and cloud shell, with customizable settings.

Create a Databricks workspace in the Azure cloud by configuring subscription, resource group, and workspace name, then launch it from the portal or its URL to access the Databricks platform.

Explore the Azure Databricks platform architecture, featuring the control plane and data plane, with a serverless workspace accessible via the UI and REST APIs.

Learn how to set up a Databricks workspace, provision a spark cluster, and use notebooks with magic commands and the Databricks utilities package to streamline data engineering workflows.

Create and configure an Azure Databricks cluster, choosing between all-purpose and single/multi-node, setting auto scaling, auto termination, worker specs, and spark runtime, then review cost estimates and cluster logs.

Learn to create and name Databricks notebooks, set Python as default, attach to a running cluster, write and run code in cells, and use export, revision history, and comments.

Explore Databricks notebook magic commands to mix languages in a single notebook, switch between Python, SQL, and Scala, and use Markdown, filesystem, and shell magics for efficient data work.

Explore Azure Databricks utilities, including dbutils packages, the file system, notebook, and widget utilities, to manage dbfs, mount cloud storage, compose modular notebooks, and pass parameters for scalable data workflows.

Learn to access data with the Databricks file system (dbfs), explore the data directory structure, create subdirectories and files, and mount Azure Data Lake Storage to dbfs.

Explore the Databricks File System (Dbfs), a distributed file system that lets you access cloud storage as a directory, and learn about the default root storage and mounting Azure storage.

Databricks auto-creates a cloud storage account for your workspace as the dbfs root; learn to access it with the dbfs file system and mount your own storage.

Create an Azure storage account with ADLS Gen2, mount its container to the dbfs root in Databricks, and secure access with a service principal and Azure Vault.

Learn Unity Catalog and metadata management in the Databricks cloud. Set up a cloud storage layer and apply fine-grained access control for lakehouse and data lake projects.

Introduce Databricks Unity Catalog as a centralized metadata store with catalogs, environments and projects, databases, tables, views, and volumes, plus external locations, storage credentials, and fine-grained access control.

Learn to set up unity catalog and a metastore within Databricks on Azure, linking storage accounts, access connectors, and workspaces to enable data governance and fine-grained access.

Learn to provision corporate and domain users into Databricks Unity Catalog using SIM provisioning, Azure Active Directory, and user groups to manage privileges and access.

Explore how to create and manage Unity Catalog objects, assign privileges with grant and revoke, and use external locations, volumes, and storage credentials in Databricks.

Explore Delta Lake and Delta Tables within the Spark and Databricks context, study versioning, time travel, schema validation and evolution, and delta table optimization through demos and examples.

Discover delta lake, an open source storage framework between processing engine and cloud storage, enabling acid transactions, delete/update/merge, schema enforcement, data versioning with time travel, and streaming and batch unification.

Learn three delta table creation methods in Databricks: DDL create table, dataframe write with save as table, and the delta table builder API; note schema implications.

Create an azure external location with a storage credential, and save a dataframe as a delta table at the external location so other teams can access the data.

Learn to read delta tables with spark sql and the data frame api, including reading delta format data from an external location and creating external tables.

Master delta table operations in Spark, including delete, update, and merge, with Spark SQL and Delta table API in Python.

Explore delta table versioning and time travel to query older states, restore data, and audit changes using spark sql, as of a given version or timestamp.

Convert a partitioned parquet data set to a delta data set in place with the convert to delta command, enabling in-place migration and delta log creation.

Understand delta table schema validation, enforcing data types and column names on inserts, including handling data type mismatches, and compare insert by position, merge, overwrite, and data frame append.

Explore delta table schema evolution, comparing manual and automatic approaches to accommodate new columns, using alter table, insert, and merge with data frame writes for safe ingestion.

Explore how delta tables store data as parquet files with a delta log and versioned json metadata, and how deletes create new data files for each version.

Delves into delta table utilities such as vacuum, history, and reorg, with dry run and retention handling, column mapping for drop column, and optimize with z-order for balanced files.

Explore incremental data ingestion in lakehouses, covering architecture and use cases, then learn copy command, spark streaming, and auto loader for ingestion with manual and automatic schema evolution.

Explore lake house medallion architecture and learn how to ingest data into bronze layer via staging areas, using copy, Spark Structured Streaming, or Databricks Auto Loader to build delta tables.

Learn to use copy into to ingest landing zone data into a bronze table with a fixed schema, and apply manual schema evolution to handle new columns.

Master Databricks copy into with automatic schema evolution to ingest csv data from a landing zone into a schema-less delta table, inferring and merging schema on the fly.

Ingest streaming data from a landing zone into a bronze delta table using Spark Structured Streaming, enabling automatic schema evolution and incremental ingestion across new files.

Explore Databricks Auto Loader, a cloud-native, spark streaming framework that efficiently ingests new files from cloud storage with incremental listing, optimized reads, and optional data landing notifications.

Learn how autoloader ingests data from a landing zone into a bronze delta table, with automatic schema evolution, inferred types, schema hints, and rescued data for errors.

Databricks Delta Live Table, a declarative ETL framework for building pipelines with two steps: declarative code and a Databricks Workflow UI, delivering data quality dashboards, lineage, and monitoring.

Build delta live tables pipelines from landing zone to bronze, silver, gold layers with incremental processing, apply QCD type two, implement CDC with merge for UK 2022 daily sales report.

Implement a Databricks DLT pipeline to ingest landing-zone CSVs into bronze, cleanse with quality checks, and apply SCD type 2 for customers and CDC for invoices.

Learn to create and schedule a delta live table pipeline using the UI, connect your code from workspace or repo, and run it against Unity Catalog or Hive metastore.

Learn to build delta live tables pipelines in python, creating bronze raw tables, cleaning with data quality, and silver scd type 2 merges, plus daily materialized views for final analytics.

Explore how to connect the Databricks workspace to source control using Databricks repos, clone and push code, manage branches, and collaborate on notebooks with Git-based systems.

Learn how to automate Databricks rest api-based deployment of workflow jobs across environments using ci cd, by copying json definitions from the ui and using access tokens.

Learn how the Databricks CLI wraps the REST API to automate Databricks tasks from the terminal, including creating jobs with inline JSON and parsing output with jq.