Master of Databricks Asset Bundle (DataOps)

Explains the CI/CD pipeline from code push to an artifact repository through test, build, and packaging, and contrasts continuous delivery with continuous deployment while highlighting early failure detection.

Master the ci/cd workflow for Databricks with asset bundles, version control, and pipelines, deploying yaml-based configurations and Terraform-managed resources across dev, staging, and production.

Learn to install and configure the Databricks CLI for Databricks asset bundle deployments, set up azure databricks, generate and use a token, and verify clusters.

Initialize your first Databricks asset bundle to auto-generate templates, deploy code and configuration as code, and set up a Databricks job across dev and prod environments.

Master the Databricks asset bundle CLI commands to validate configurations, view bundle summaries, deploy to dev or prod, open jobs in the UI, and safely destroy resources within CI/CD pipelines.

Explore how to locate yaml asset bundle parameters by inspecting the json schema linked from the yaml and identify triggers and time unit options.

Extend a Databricks asset bundle by adding multiple tasks in init project job yaml, sharing the same cluster, and deploying with depends on to chain notebooks or run in parallel.

Configure Databricks notebook dependencies to run notebooks in parallel and run a third notebook only after prior ones finish, using run if options such as all succeeded or all done.

Learn to set up a databricks asset bundle project with git integration, choosing GitHub for version control, and push changes from local to remote through two setup approaches.

Learn how to initialize and version a Databricks asset bundle by first creating and configuring a GitHub repository, then cloning, initializing a Databricks bundle, and branching workflows.

Integrate GitHub with Databricks to synchronize notebooks and source code using personal access tokens, create a GitHub folder, and manage deployments via local environments and yaml workflows.

Explore substitution or internal variables in Databricks asset bundles, learn how to override and retrieve variables, and validate production targets with a JSON output.

Define variables in the Databricks YAML file, reference them in notebooks with a default value, and override per target using complex variables to build modules.

Demonstrates overriding bundle variables during execution with the Databricks bundle CLI, using --var to replace defaults (for notebooks) with environment-specific values like hello prod.

Learn how to use complex variables to parameterize cluster configurations for multiple tasks in a Databricks job, organizing clusters in YAML files and referencing them in tasks with bundle validate.

Learn how retrieval variables in Databricks fetch an existing cluster id to run tasks on a shared cluster, and understand cost considerations of keeping a running 24/7 generic cluster.

Define parameters at the job level or task level so all tasks can consume them; pass static or dynamic values via the Databricks yaml and json schema.

Explore how to pass parameters between tasks in a Databricks asset bundle workflow by using dynamic task values, dbutils jobs, and notebook-level parameters to share and update data.

Learn to pass parameters to Databricks jobs using Spark config and environment variables, including the partition date, and read them in notebooks or Python code.

Learn to chain multiple Databricks jobs and pass parameters between them. This lesson shows triggering a second job from the first and transferring values via job parameters.

Automate your Databricks asset updates with an on push pipeline that validates, builds, and deploys changes via GitHub actions. Manage environments and secrets for secure, repeatable deployments.

Implement a pull request trigger in your GitHub CI/CD pipeline to validate changes before merging into main, and enable status checks like deploy bundle and run pipeline update.

Learn to deploy a Databricks Asset Bundle via an Azure DevOps pipeline using a token and a service principal, with yaml configuration and GitHub integration.

Deploy your Databricks asset bundle via an Azure DevOps pipeline secured by a service principal, onboarding it in Databricks accounts.

Configure unity catalog for Azure Databricks by creating a metastore in a storage account with hierarchical namespace, connecting via an access connector, assigning it to the workspace, and provisioning schemas.

Ingest NYC taxi data into the Unity Catalog by bulk copying from Azure Open Data Storage using AzCopy, then configure a metastore volume and permissions.

Set up a Databricks asset bundle for the yellow NYC project, initialize a cluster and job, validate and deploy, then connect to GitHub and prepare notebooks for the next lesson.

Learn how the dab_demo job chains tasks: ingestion data, filter data, and visualization data, passes job-level parameters via JSON, uses Unity Catalog, and configures a Databricks YAML workflow for visualization.