Azure Databricks End-to-End: Unity Catalog DABs CICD - 2026

Create an Azure Databricks premium workspace in the Azure portal by selecting subscription, resource group, region, and a name, and enable Unity Catalog before deployment.

Explore the Azure Databricks workspace UI, learn to navigate notebooks, clusters, jobs, SQL workspaces, and pipelines, and use the advanced search to manage your end-to-end Databricks workflow.

Explore how Azure Databricks provisions a compute cluster by creating virtual machines, a VNet, and a storage account within a managed resource group, and understand cluster lifecycle and cost implications.

Master Databricks magic commands to run multiple languages in a single notebook, switch cell languages with % commands, and explore file system and other magic utilities in the community edition.

Learn to parameterize notebooks with dbutils widget utilities, enabling environment-aware ci/cd across dev, uat, and prod using combo boxes, drop-downs, multi-selects, and text widgets.

Demonstrate notebook utilities in Azure Databricks, including running one notebook from another with dbutils.notebook.run, using widgets for parameters, and returning exit values.

Explore how Delta Lake blends reliability with data lake flexibility to transform storage and management. Experience its features through practical notebook implementations, achieving robust, unified asset transactions and schema evolution.

Trace the evolution from traditional data warehouses to lakehouse architecture, blending data lake flexibility with warehouse reliability. Delta Lake adds transactional logs and metadata for unified analytics.

Create a new Databricks workspace and storage account for Delta Lake, compare standard and premium tiers, and explore Delta Live Tables and SQL warehouses in the new environment.

Learn to access an Azure data lake storage from a Databricks workspace with a service principal, assign storage blob data contributor, and run a notebook to read a CSV.

Learn to create a delta lake by swapping parquet for delta format in Azure Data Lake storage, and explore the delta log and transaction log implications.

Explore how the delta log and json transaction log record commits, schemas, and file operations in Delta Lake, including overwrite, add, time travel, and versioning.

Upload key files to a new schema eval directory to prepare for upcoming lectures, download and place the files in the folder, and use them to guide the next lessons.

Learn schema enforcement in Delta Lake, a gatekeeper that validates writes against the table schema, rejects extra columns and data-type mismatches, and allows nulls for missing fields.

Learn how delta lake schema evolution lets you adapt a table to evolving data by adding new columns with merge schema, or overwrite when data types change in Azure Databricks.

Explore Delta Lake time travel and versioning to audit changes, rollback accidental writes, and restore previous table states with describe history and version as of.

Master the vacuum command in Delta Lake to delete obsolete parquet files, using dry run to preview deletions, and manage retention with seven days by default or custom hours.

Explore Unity Catalog in Databricks, its governance model and object model, simulate a permissions environment, and learn how to assign Unity Catalog privileges for real-time projects.

Create a Databricks access connector as a managed identity to grant Unity Catalog access to Azure Data Lake Storage with the storage blob data contributor role, enabling metastore access.

Explore how cluster pools pre-provision idle virtual machines to speed Databricks cluster creation, reducing wait times while balancing idle infrastructure costs with performance.

Learn to create and configure an Azure Databricks cluster pool, set minimum idle and maximum capacity, apply termination times, and enforce pool-based compute with policies and permissions.

Create a dev catalog under Unity Catalog to organize bronze, silver, and gold schemas by environment. Learn how metastore, projects, and permissions govern centralized catalog creation across workspaces.

Master Unity Catalog privileges in Azure Databricks, granting use catalog and select permissions, managing ownership, and using the three level namespace (catalog, schema, table) to access governed objects.

Assign roles to Databricks access connector: storage blob data contributor, storage queue data contributor, storage account contributor, event grid contributor, and event subscription contributor; verify access before creating external location.

Create storage credentials and external locations in Unity Catalog, test the connection, and apply fine-grained access controls to securely access data across storage containers.

Explore Unity Catalog managed tables with delta format and 30-day retention, and external tables backed by external locations and storage credentials.

Explore how volumes in Unity Catalog enable governance over non-tabular data by registering a volume under a catalog and schema that points to an Azure Data Lake path.

Explore the fundamentals of Spark structured streaming, including reading and writing streams, checkpoints, and autoloader, while previewing concepts like watermarking and windowing essential for project implementation.

Explore Spark structured streaming basics by visualizing an incoming data stream from IoT devices and app logs, and learn how continuous data appends to an unbounded destination table.

Discover the supported sources and sinks for Spark structured streaming, including file, Kafka, socket, and table sources, plus file, Kafka, console, for each, and table sinks.

Understand why the Databricks community edition drops databases when a cluster terminates, and learn to recreate schemas and tables using the hive metastore location with repeatable scripts.

Understand the default trigger, the processing time trigger, and the available now trigger, including checkpoint behavior and micro-batches in structured streaming.

Leverage autoloader with cloud files to infer csv schemas, using schema location and infer schema types to true. Explore schema hints and evolution in streaming reads on dbfs.

Explore practical schema evolution in Azure Databricks, using schema evolution modes like rescue, add new columns, fail on new columns, and none, to handle unknown fields in streaming data.

Explore the medallion architecture and its value in data engineering, and learn to implement it in an Azure Databricks project with Unity Catalog CICD using a roads dataset.

Explore the end-to-end medallion architecture in Azure Databricks, building a landing zone and bronze, silver, and gold layers in data lake storage with incremental ingestion and Unity Catalog governance.

Understand raw traffic and roads datasets, learn to build a data dictionary, and analyze vehicle counts by type, direction, and road attributes in an end-to-end databricks workflow.

Set up external locations, containers, bronze tables in Azure Data Lake and Databricks to accept landing data, establish schemas under catalog, and implement reusable PySpark code across dev, uat, prod.

Ingest landing zone data into the bronze layer by creating two tables, raw traffic and raw roads, using spark sql and parameterized functions for robust ingestion.

Ingest data from landing zone folders in Azure Data Lake into bronze tables using Databricks notebooks. Map raw roads to raw roads table and raw traffic to raw traffic table.

Ingest raw_traffic data from the landing zone into a delta bronze table using auto loader, with explicit schema, checkpoints, and batch trigger for end-to-end Databricks workflow.

Demonstrate that only newly added records are processed during incremental loading from landing to bronze and silver, using Spark Structured Streaming and Delta Lake to prove selective transformation.

Create a common notebook to store reusable variables (checkpoint, landing, bronze, silver, gold) and functions (remove duplicates, remove nulls) and access them across notebooks via the %run command.

Read the bronze roads data in Azure Databricks, clean it by removing duplicates and nulls, derive road category name and road type with PySpark, then write to silver roads table.