Databricks Asset Bundles 101

In this lecture, you will learn how to download and install the Databricks CLI on Windows. We will go step by step through selecting the correct version, downloading the installation file and extracting it to the user folder. This is the first step in setting up the Databricks CLI for managing Databricks Artefacts efficiently.

What students will be able to do after this lecture:

• Download the correct version of Databricks CLI

• Extract and organize the CLI files in a user folder directory

• Complete the installation process for future setup

In this lecture, you will learn how to add the Databricks CLI path to your system's environment variables on Windows. We will walk through accessing the system settings, locating the correct user folder path, and appending it to the system's PATH variable. This step is essential to ensure that the Databricks CLI can be run from any command prompt window on your system.

What students will be able to do after this lecture:

• Access and modify system environment variables on Windows

• Add  Databricks CLI folder to the system PATH

• Verify that the Databricks CLI is recognized in the command line interface

In this lecture, you will learn how to verify that the Databricks CLI has been installed correctly on your Windows system. We will demonstrate how to run a simple command in the command prompt to confirm that the CLI is recognized and functioning properly. This ensures your setup is complete and ready for configuration.

What students will be able to do after this lecture:

• Open and use the command prompt to interact with the CLI

• Run the databricks --version command to confirm installation

In this lecture, you will learn how to generate a Personal Access Token (PAT) in the Databricks workspace. We will guide you through navigating to the user settings and creating a new token, which will later be used to authenticate the Databricks CLI with your account.

What students will be able to do after this lecture:

• Access user settings in the Databricks workspace

• Generate a new Personal Access Token

Authenticate with Databricks CLI

In this lecture, you will learn how to authenticate with your Databricks workspace using the Databricks CLI.
We’ll walk through the databricks configure command, explain how to use a Personal Access Token, and validate your setup using databricks auth env. This authentication step is essential before you can run any Databricks CLI or Asset Bundle commands.

What students will be able to do after this lecture:

• Authenticate with a Databricks workspace using the Databricks CLI

• Use a Personal Access Token securely for CLI access

• Locate and understand the .databrickscfg file and CLI profile behavior

• Validate CLI authentication using databricks auth env

In this lecture, you will learn how to initialize a sample Databricks Asset Bundle using the Databricks CLI. We’ll walk through using the databricks bundle init command to generate a scaffold project, explore the key files created (like databricks.yml), and understand how this structure forms the foundation for managing and deploying Databricks assets.

What students will be able to do after this lecture:

• Run the databricks bundle init command to generate a new bundle

• Identify key files and folders in a bundle structure

• Understand the purpose of databricks.yml in managing Databricks assets

In this lecture, you will learn how to validate your bundle project using the databricks bundle validate command.

We’ll demonstrate how validation helps catch issues early in the development cycle—such as syntax error or missing configuration fields—and ensure your project is ready for deployment. You’ll also see how warnings and errors are displayed, and what steps to take when they appear.

What students will be able to do after this lecture:

• Run the databricks bundle validate command with a specific deployment target

• Understand how validation checks the correctness of your bundle configuration

• Identify and resolve common warnings or errors before deployment

• Build confidence in the readiness of your bundle for staging or production environments

In this lecture, you will learn how to deploy your bundle project to a development environment using the databricks bundle deploy -t dev command.

We’ll walk through the end-to-end process of preparing your local bundle project, ensuring your databricks.yml file is correctly configured, and pushing your resources—like notebooks, jobs, and pipelines—into the specified development workspace. You’ll also see how the CLI output confirms what was deployed, helping you verify everything landed in the right place.

What students will be able to do after this lecture:

• Run the databricks bundle deploy -t dev command to deploy resources

• Understand how the CLI uses the dev target defined in databricks.yml

• Verify successful deployments using CLI feedback and workspace inspection

• Build confidence deploying safely to a development environment before promoting to staging or production

In this lecture, you will learn how to deploy your Databricks bundle to a staging environment using the databricks bundle deploy -t staging command.

We’ll guide you through how to configure a separate staging profile, update your databricks.yml file with the correct target settings, and push your bundle resources to the designated staging workspace. You'll see how this deployment step fits into a CI/CD workflow, and how to confirm that notebooks, jobs, and configurations have been deployed successfully.

? What students will be able to do after this lecture:

• Run the databricks bundle deploy -t staging command for staging deployments

• Configure a dedicated staging target in the databricks.yml file

• Understand how staging deployments help separate development from testing environments

• Verify and validate that resources are correctly deployed to the staging workspace

• Prepare confidently for production releases by testing in a controlled environment

In this lecture, you will learn how to run a job that’s defined within your Databricks bundle using the databricks bundle run command.

We’ll demonstrate how this command allows you to trigger jobs locally via Databricks CLI. You’ll see how it connects with the Databricks workspace, how to monitor job execution, and how to troubleshoot directly from the CLI or workspace UI.

What students will be able to do after this lecture:

• Use the databricks bundle run <job_name> command to trigger jobs from the CLI

• Understand how the bundle context (targets, config, resources) is used during job execution

• Monitor job progress via CLI output and Databricks workspace UI

• Inspect logs and task statuses for troubleshooting

• Confidently run and test jobs before scheduling or CI/CD automation

In this lecture, you will learn how to clean up your deployed Databricks resources using the databricks bundle destroy command.

We’ll walk through how this command removes jobs and other assets associated with a specific deployment target, and how it helps keep your workspace organized. You’ll see how to review the destruction plan, confirm deletions, and clean up remote files from the workspace.

What students will be able to do after this lecture:

• Run the databricks bundle destroy -t <target> command to remove deployed resources

• Understand the destruction plan generated before deletion

• Safely confirm and execute resource cleanup

• Delete remote bundle files from the workspace root path

• Use bundle destroy to keep development and staging environments clean and manageable

In this lecture, you will learn how to manage and structure your code using a clear Git branching strategy within Azure DevOps Repos.

We’ll walk through how feature branches, dev, and main branches support safe development and deployment workflows. You’ll see how pull requests and environment-based promotion fit into a typical CI/CD process. This strategy helps your team collaborate efficiently, reduce merge conflicts, and maintain production stability.

What students will be able to do after this lecture:

• Create and manage feature branches for isolated development

• Understand the purpose of dev and main branches in CI/CD workflows

• Use pull requests and branch policies to enforce quality control

• Align Git branching with deployment stages in Databricks bundle pipelines

• Promote changes safely through development, staging, and production environments

In this lecture, you will learn how Databricks bundles integrate with a CI/CD pipeline to enable structured and automated deployments across development, staging, and production environments.

We’ll walk through the full deployment lifecycle — from triggering deployments in development, to automated promotion in staging, and final rollout to production. You'll see how Git branching, environment-specific targets, and CI/CD pipelines work together to ensure consistency, reliability, and control in your Databricks projects.

What students will be able to do after this lecture:

• Understand the end-to-end CI/CD flow for Databricks bundle deployments

• Deploy bundles to development, staging, and production environments using environment targets

• Trigger automated deployments through Azure DevOps pipelines

• Maintain clean and controlled release cycles using bundle-based workflows

In this lecture, you will learn how to integrate Git with Databricks to enable version-controlled, collaborative development directly within your workspace.

We’ll walk through the entire Git integration process — from generating a Personal Access Token in Azure DevOps, to configuring Git credentials in Databricks, and cloning remote repositories as Git folders. You’ll also learn how to create feature branches and manage your work in isolated development streams.

What students will be able to do after this lecture:

• Connect Databricks to Azure DevOps or other Git providers using secure authentication

• Clone and manage Git repositories inside Databricks workspaces

• Create and work with feature branches for isolated development

• Collaborate effectively using Git folders and best practices for branch management

In this lecture, you will learn about the Azure DevOps pipeline stages involved in deploying Databricks asset bundles, focusing on the Build Artifacts, Dev, Staging, and Production stages.

We’ll explore how each stage is defined, what role it plays in the deployment process, and how they are triggered — whether manually or automatically — to provide a controlled and reliable release workflow.

What students will be able to do after this lecture:

• Understand the purpose of the Build Artifacts, Dev, Staging, and Production stages in an Azure DevOps pipeline

• Learn how each stage is triggered, including manual triggers and branch-based automatic triggers

• Gain insight into how controlled deployments support consistent and repeatable Databricks bundle releases across environments

• Prepare to work with pipeline YAML configurations in upcoming lessons

In this lecture, you will learn how to structure and configure an Azure DevOps pipeline to deploy Databricks asset bundles across multiple environments.

We’ll walk through the key pipeline stages — Build Artifacts, Dev, Staging, and Production — and explore how each stage is defined in YAML. You'll gain a clear understanding of how manual and automatic triggers work, including how feature branches are handled using conditional logic.

This lesson focuses on giving you the confidence to understand and customize pipeline behavior before moving into actual deployment.

What students will be able to do after this lecture:

• Identify the purpose of each pipeline stage: Build, Dev, Staging, and Production

• Understand how to configure manual and automatic triggers using YAML conditions

• Interpret and navigate Azure DevOps YAML files for stage configuration

• Recognize how the Databricks CLI is used to validate and deploy bundles

• Understand the role of secure environment variables for service principal authentication

In this lecture, you’ll learn how to manually trigger the Dev stage in an Azure DevOps pipeline using a feature branch, and verify the resulting deployment in the Databricks Dev workspace.

We’ll walk through selecting the pipeline, choosing the correct branch, running only the necessary stages, and confirming that the Databricks asset bundle — including jobs and notebooks — is successfully deployed.

You’ll also gain an understanding of how manual trigger conditions are defined in YAML, and why this setup allows for controlled and safe deployments during development.

What students will be able to do after this lecture:

• Run an Azure DevOps pipeline manually from a feature branch

• Understand how manual trigger conditions are configured for the Dev stage

• Select specific stages to run in a multi-stage pipeline

• Verify the deployment of workflows and notebooks in the Databricks Dev workspace

• Understand how Databricks bundles integrate into CI/CD pipelines

In this lecture, you’ll learn how to trigger the Staging deployment in an Azure DevOps pipeline automatically by creating a pull request into the dev branch.

We’ll walk through the entire process — from preparing the feature branch, submitting a pull request, and merging it into dev, to watching the pipeline automatically trigger the StagingDeployment stage.

You’ll also see how the pipeline condition is defined in YAML to respond to dev branch changes, and how the Databricks asset bundle is validated and deployed to the Staging environment.

What students will be able to do after this lecture:

• Create a pull request from a feature branch to dev in Azure Repos

• Trigger a pipeline run automatically based on branch conditions

• Understand how YAML conditions are used to control stage execution

• Observe the automatic deployment of jobs and notebooks to the Databricks Staging workspace

• Gain confidence in using pull requests as a gate to promote code between environments

In this lesson, you’ll learn how changes are promoted to the Production environment using an automated, branch-based CI/CD pipeline in Azure DevOps.

We’ll walk through the process of creating a pull request from dev into main, which triggers the final stages of the pipeline. You’ll see how the pipeline first revalidates the bundle through the Staging Deployment stage before proceeding to the Production Deployment, ensuring consistency and confidence in the release process.

This lesson emphasizes automation with control — where production deployments happen only after prior validation and approval.

What students will be able to do after this lesson:

• Understand how Production deployment is triggered by merging into the main branch

• Configure branch-based conditions for gated deployments in YAML

• Trace the end-to-end CI/CD flow from staging to production

• Validate that jobs and notebooks are promoted correctly to the Production Databricks workspace

In this lesson, you'll learn how to deploy a Databricks Asset Bundle directly from the Databricks Workspace UI.

We’ll walk through the structure of a bundle project, including where to place notebooks and workflow definitions, and how the databricks.yml file ties everything together. You’ll then see how Databricks detects your bundle, shows available deployment targets, and allows you to deploy with just a few clicks.

You’ll also learn how to verify deployed jobs and notebooks inside the workspace, giving you full visibility into what gets pushed and where.

What students will be able to do after this lesson:

• Understand how Databricks detects bundles in the Workspace UI

• Deploy a bundle to a specific target (e.g., dev) directly from the UI

• Review the deployment summary before confirming

• Locate and verify deployed jobs and notebooks in the workspace

• Gain confidence in UI-based deployment workflows for Asset Bundles

In this lesson, you’ll learn how to remove previously deployed resources from your Databricks workspace using the built-in Workspace UI.

We’ll walk through how to locate a deployed bundle, use the More Options menu to trigger the bundle destroy action, and verify that jobs and notebooks have been properly removed from the workspace. This provides a clean and visual way to manage your asset lifecycle — all without using the CLI.

What students will be able to do after this lesson:

• Identify previously deployed asset bundle jobs in the workspace

• Use the Workspace UI to trigger resource cleanup with the “Delete deployed resources” action

• Understand how bundle destroy is executed behind the scenes

• Verify that workflows and notebook artifacts have been removed from the workspace

• Maintain a clean and consistent deployment environment

In this lesson, you’ll learn how to override bundle variables at runtime using the built-in Databricks Workspace UI.

We’ll walk through how to open the variable-overrides.json file using the “Configure variable overrides” option, define deployment-specific values like environment and project, and reference them in your bundle configuration for dynamic tagging and customization. This provides a flexible way to deploy the same bundle across multiple environments — without changing your core bundle code.

What students will be able to do after this lesson:

• Open and edit variable-overrides.json via the Workspace UI

• Define runtime variable overrides such as environment, project, and team

• Reference override values in bundle configuration files using ${variable_name} syntax

• Deploy bundles with environment-specific tags and logic

• Reuse a single bundle for multiple deployment targets by changing only variable values

In this demo, you'll see how Unity Catalog artefacts are deployed automatically using a Databricks Asset Bundle with the AdventureWorks dataset.

We’ll walk through the pipeline execution and show how it creates catalogs, schemas, tables, and views, along with workflows, SQL compute, and dashboards. You’ll watch the AdventureWorksDataLoad workflow orchestrate data movement from Bronze to Silver to Gold, and see how everything is deployed directly from code with no manual steps.

You’ll also learn how to inspect and verify all deployed artifacts in the Databricks workspace, giving you a clear picture of the deployment outcome.

What students will be able to do after this demo:

• Observe the deployment of Unity Catalog artefacts using the AdventureWorks dataset

• Watch pipeline execution and automated workflow orchestration

• Inspect deployed catalogs, schemas, tables, views, workflows, compute, and dashboards

• Understand how end-to-end deployment works in a live Databricks environment

• Gain confidence in reviewing automated deployments and outcomes

In this briefing, you’ll review the course materials and confirm that your Databricks environment is ready to follow along with the Unity Catalog deployment lessons using the AdventureWorks dataset.

We’ll walk through the attached unity_catalog_cicd source code and highlight what it contains, including Unity Catalog definitions, workflows, SQL assets, and CI/CD configuration. You’ll also get an overview of the environment prerequisites that must be in place before running the deployment pipeline.

This lesson sets the foundation for the hands-on setup and deployment steps that follow, ensuring you know exactly what to prepare and what will be configured next.

What students will be able to do after this briefing:

• Identify and access the provided unity_catalog_cicd course materials

• Understand the structure and purpose of the source code used in the course

• Recognize the required environment prerequisites for Unity Catalog deployment

• Confirm their Databricks workspace is ready for the upcoming setup steps

• Transition confidently into the hands-on configuration and deployment lessons

In this lesson, you’ll learn how to ensure your Databricks workspace is connected to a Unity Catalog metastore, which is required to create catalogs, schemas, and tables.

We’ll walk through verifying if a metastore is already attached to your workspace and show how to create a new metastore if needed. You’ll see how to configure key settings such as the metastore name, region, and storage location, and how to attach it to your workspace so it’s ready for catalog deployment.

This hands-on setup ensures your environment is properly configured for the upcoming Unity Catalog deployments and CI/CD pipeline exercises.

What students will be able to do after this lesson:

• Verify if a Unity Catalog metastore is attached to their Databricks workspace

• Create a new Unity Catalog metastore if one does not exist

• Configure metastore settings, including name, region, and storage options

• Attach the metastore to the Databricks workspace

• Prepare the workspace for deploying catalogs, schemas, tables, and views

In this lesson, you’ll learn how to create and configure a service principal that will be used for CI/CD automation and Unity Catalog deployments.

We’ll walk through creating a service principal in Databricks, assigning the appropriate permissions, and preparing it for use in automated workflows. You’ll also see how the service principal integrates with role-based access control to securely manage catalog objects and deployment tasks.

You’ll then review how the service principal credentials are stored securely in Azure Key Vault, ensuring that authentication details are never hard-coded and can be safely used by the CI/CD pipeline.

What students will be able to do after this lesson:

• Create a service principal in Databricks

• Understand the role of a service principal in CI/CD and Unity Catalog management

• Assign appropriate permissions to control who can use and manage the service principal

• Store client ID and client secret securely in Azure Key Vault

• Prepare the environment for automated, secure deployments

In this lesson, you’ll learn how to create role-based groups to manage access and permissions in Unity Catalog.

We’ll walk through creating two key groups: adb-metastore-admins for full metastore management, and dev-adb-catalog-owners for full ownership of catalog artifacts, including catalogs, schemas, tables, and views.

You’ll also see how to add members, including the service principal databricks_workflow_sp_dev for automated deployments, and optionally yourself for demo purposes. Finally, you’ll configure which admin groups can manage these groups, ensuring secure and governed access.

What students will be able to do after this lesson:

• Create role-based groups for Unity Catalog access management

• Understand the responsibilities of metastore admins and catalog owners

• Add members to groups, including service principals and users

• Assign group-level permissions and governance controls

• Prepare the workspace for secure, automated CI/CD deployment of Unity Catalog artifacts

In this lesson, you’ll learn how to set up the Azure Data Lake Storage account used by Unity Catalog and the data pipeline.

We’ll walk through creating the required storage containers, including a landing container for raw data ingestion and a unitycatalog container for managed data and metadata. You’ll also learn how to grant the Databricks managed identity the proper permissions to securely access these containers.

This setup ensures that the storage layer is properly organized, secure, and ready to support catalog-managed tables, views, and automated CI/CD deployment.

What students will be able to do after this lesson:

• Create storage containers for raw data and Unity Catalog managed data

• Understand the role of the landing and unitycatalog containers

• Upload source data files into the landing container following the required folder structure

• Grant the Databricks managed identity Storage Blob Data Contributor access to the containers

• Grant the Databricks managed identity Storage Blob Delegator access to the storage account

• Prepare the storage environment for Unity Catalog and CI/CD pipelines

In this lesson, you’ll learn how to create a storage credential in Unity Catalog to securely access Azure Data Lake Storage.

We’ll walk through using the Azure Managed Identity to authenticate with the storage account, assign a descriptive name for the credential, and configure it so it can be reused by external locations and catalogs.

You’ll also learn how to assign permissions on the storage credential, controlling who can use it and which Databricks workspace can access it. This ensures secure, role-based access for your development and CI/CD environments.

What students will be able to do after this lesson:

• Create a storage credential using Azure Managed Identity

• Assign a meaningful name to the storage credential for easy reference

• Understand how storage credentials allow secure, keyless access to storage

• Grant permissions to users or admin groups to manage and use the credential

• Assign the storage credential to a specific Databricks workspace for controlled access

• Prepare the environment for creating external locations and deploying Unity Catalog artifacts

In this lesson, you’ll learn how to create and configure external locations in Unity Catalog to securely connect Databricks with your underlying storage account.

We’ll walk through setting up two external locations: one for raw source data and another for managed Unity Catalog data. You’ll see how to link each external location to the appropriate storage container using a storage credential, and how to control access through permissions and workspace assignments.

By the end of this lesson, you’ll understand how external locations enable secure, governed access to data, and how they form a critical foundation for managing Bronze, Silver, and Gold tables in a Unity Catalog–enabled environment.

What you’ll learn in this lesson:

• How to create an external location for raw data ingestion

• How to create an external location for Unity Catalog–managed data

• How to associate external locations with storage credentials

• How to grant permissions and restrict workspace access

• How external locations fit into a secure, governed data architecture

In this lesson, you’ll learn how to securely manage sensitive credentials in Databricks by creating an Azure Key Vault–backed secret scope.

You’ll walk through the process of linking Azure Key Vault to Databricks, allowing secrets such as service principal credentials to be stored and accessed securely without exposing them in code or configuration files.

This lesson covers how to create the secret scope, configure the required Key Vault settings, and verify that the scope is working correctly using the Databricks CLI. By the end of this lesson, you’ll understand how to safely manage secrets and integrate them into your Databricks workflows and CI/CD pipelines.

What you’ll learn in this lesson:

• How to create a Key Vault–backed secret scope in Databricks

• How to connect Databricks to Azure Key Vault

• How to securely store and reference secrets

• How to verify secret scope creation using the Databricks CLI

In this lesson, you’ll learn how to securely store all the sensitive credentials and configuration values required for your Databricks deployment in Azure Key Vault.

We’ll walk through creating each secret needed for the project, including service principal credentials, Databricks workspace information, Unity Catalog configuration, and storage account details. These secrets will be referenced in Databricks through the Key Vault–backed secret scope, allowing pipelines and notebooks to access them securely without exposing sensitive information in code.

By the end of this lesson, you’ll understand how to:

• Create secrets in Azure Key Vault for use in Databricks

• Store service principal credentials and workspace configuration securely

• Reference secrets in Databricks pipelines and workflows through a Key Vault–backed secret scope

• Follow best practices for secret management and secure automation

In this lesson, you’ll learn how to set up the project source code in Azure DevOps and prepare a dedicated feature branch for Unity Catalog CI/CD deployment.

We’ll walk through creating a new repository, cloning it to your local machine, adding the provided course source code, and committing it to a new feature branch called features/unitycatalogcicd.

This lesson also covers verifying the repository status and pushing changes to Azure DevOps, ensuring that the repository is correctly configured and ready for automated deployment.

What you’ll learn in this lesson:

• How to create a new repository in Azure DevOps for the course source code

• How to clone the repository locally and organize files

• How to create and switch to a feature branch

• How to stage, commit, and push changes to the repository

• How to prepare a version-controlled environment for CI/CD deployments

In this lesson, you will update the databricks.yml to specify both the target deployment workspace and the service principal used for authentication.

You’ll learn how to:

• Update the workspace host to point to your development Databricks environment

• Update the service principal name to use the Application (Client) ID of the service principal created earlier

• Ensure the CI/CD pipeline deploys resources securely and to the correct workspace

  
  

In this lesson, you will set up the Azure DevOps pipeline for deploying Databricks resources, configure a Key Vault–linked variable group, and create a DEV environment with appropriate permissions.

You’ll learn how to:

• Create a new Azure DevOps pipeline using the existing azure-pipelines.yml file from the features/unitycatalogcicd branch

• Link an Azure Key Vault to a DEV variable group and add required secrets as pipeline variables

• Grant the pipeline permission to use the variable group securely

• Create a DEV environment in Azure DevOps and assign the pipeline access permissions

In this lesson, you'll see the full deployment of Unity Catalog artefacts and associated data pipelines using the AdventureWorks dataset.

We’ll start by triggering the Azure DevOps pipeline, which deploys all necessary components to the Dev Databricks environment. You’ll watch as the pipeline creates the adventurework_dev catalog, schemas, tables, views, workflows, and the SQL warehouse required to support dashboards and analytics.

Next, we’ll verify that the code artifacts are deployed in the workspace and inspect the deployed Unity Catalog objects. You’ll see the workflows provisioned automatically, and then we’ll run the AdventureWorkDataLoad workflow to process raw data into the Bronze, Silver, and Gold layers.

Finally, we’ll review the Sales Insight Dashboard, powered by the Gold tables and the serverless SQL warehouse, to confirm that the processed data is correctly visualized.

What students will be able to do after this lesson:

• Trigger an Azure DevOps CI/CD pipeline to deploy Databricks resources.

• Inspect the deployed Unity Catalog artifacts, including catalogs, schemas, tables, and views.

• Verify that workflows have been provisioned and understand their role in data processing.

• Run workflows to process raw data into Bronze, Silver, and Gold tables.

• Explore the SQL warehouse and dashboards to confirm that analytics are functioning correctly.

• Gain confidence in observing and validating an end-to-end Databricks deployment using Asset Bundles.