CI/CD with Databricks (Declarative Automation Bundles)

Set up Azure Databricks workspaces for dev, test, and prod in dedicated resource groups, configure premium region UK South, and align the metastore with Unity catalog across environments.

Install and set up Visual Studio Code as the required IDE, install the Databricks extension, and explore the editor’s activity bar, search, source control, and debug pane.

Install a compatible Java version (8, 11, or 17) to run Spark locally, guided by the Spark–Java compatibility matrix; check java -version and Databricks runtime LTS 15.4 for production workflows.

Install Python 3.11 to match the Databricks runtime 15.4, and verify the interpreter in Visual Studio Code to ensure major and minor version compatibility; download from python.org if needed.

Install and verify the Databricks CLI using curl across Linux, Mac, and Windows, with sudo permissions when needed, and confirm the installation with databricks --version.

Configure and authenticate the Databricks CLI in VS Code using host, multiple profiles (dev and test), and personal access tokens, then verify with auth commands and locate the Databricks CFG.

Learn to manage Databricks clusters with the Databricks CLI and REST API, creating, listing, and deleting clusters using JSON templates and UI-guided workflows in VS Code.

Initiate a Databricks Asset Bundle project using the default python template, configure the bundle with YAML, and explore the project structure, including the Databricks YAML file and resources.

Validate the Databricks asset bundle, review workspace paths and targets, then deploy with Databricks bundle deploy to the dev target and verify in the workspace.

Deploy a Databricks bundle to dev, test, and prod targets by updating YAML configuration, hosts, and shared folders, while managing profiles and tokens in a CI/CD context.

Learn to define and deploy a databricks workflow job from asset bundles, including yaml templates, notebook tasks, and deployment modes with prefixes.

Modularize Databricks YAML configurations by moving resources into separate files in the resources folder, using the include key to deploy with Databricks bundle deploy, and resolve relative notebook paths.

Destroy the Databricks bundle to permanently delete deployed assets such as jobs, pipelines, and artifacts in the dev target, after a confirmation prompt, and specify another target with -t.

Databricks tracks asset bundle deployments with a manifest directory, comparing local assets to the last deployed state and redeploying only changed files for incremental updates.

Create a unity catalog enabled, single-node Databricks cluster for local development with asset bundles, ds3 v2 15.4 LTS, and ensure Python and Java versions, terminate after ten minutes of inactivity.

Connect Visual Studio Code to a Databricks compute cluster via the Databricks extension, run notebooks as jobs, and learn why individual cells don’t execute on the cluster without Databricks Connect.

Install Databricks Connect in a Python 3.11 virtual environment to run notebook cells and scripts on a remote Databricks cluster, selecting the virtual environment as the kernel.

Create a Databricks session to run scripts in the terminal with Databricks Connect, configure host, token, and cluster ID, and differentiate it from a PySpark spark session.

Learn to choose the Python interpreter in VS Code, synchronize it with the active virtual environment, and run scripts on a Databricks cluster using UI or terminal.

Develop a medallion-architecture data pipeline for Jersey City bike data from March 2025 using Databricks asset bundles; start notebooks, refactor into Python scripts, and convert to Delta Live Tables pipelines.

Create catalogs across dev, test, and prod with landing, bronze, silver, and gold schemas, then upload the Citi Bike data into each catalog's landing schema volume.

Map the city bike ETL project’s codebase, with four notebooks across bronze, silver, and gold layers, and a reusable src package for domain and utilities.

Walk through the bronze notebook, define the city bike schema, read data from the landing volume, add a metadata map, and write to the bronze table with overwrite.

Walk through the silver notebook, compute trip duration in minutes, extract trip start dates using citibike utils and datetime utils, rearrange columns, and write to silver.jsi_citibike in Databricks.

Walk through gold notebooks in a Databricks asset bundle, reading silver layer, computing max, min, and avg trip duration and total trips. Write to gold schema and show ETL lineage.

Configure Databricks Connect imports by appending the project root to sys.path, enabling local modules like datetime utils and citibike utils to import correctly from the src folder.

Configure a Databricks job to run bronze, silver, and gold Citi Bike notebooks as dependent tasks using a modular yaml asset bundle and a shared job cluster.

Configure base parameters for notebook tasks using dynamic references (job id, run id, task id, processed timestamp), retrieve with dbutils.widgets.get, and apply them in the ETL workflow.

Define a dynamic catalog variable in the Databricks YAML bundle, pass it as a base parameter to notebooks, and override it for dev, test, and prod deployments.

Override the catalog variable in the databricks configuration file with target-specific values for dev, test, and prod. Deploy to targets to see city bike underscore dev, test, and prod.

Store and reference complex variables to manage nested cluster configurations in Databricks asset bundles, enabling centralized, dynamic overrides of compute settings across tasks and environments.

Learn how to use substitution and lookup variables in Databricks asset bundles to retrieve object IDs (like clusters) by name, and apply them in YAML deployments with validation and cleanup.

Refactor Citibike ETL notebooks into Python scripts and build a job that runs each script with a Python script task, highlighting parameter handling differences from notebook tasks.

Refactor city bike ETL pipeline from notebooks to Databricks asset bundle Python scripts, deploy via YAML job, and run Citibike ETL job to rebuild bronze, silver, and gold tables.

Package your Python code as a wheel using setuptools and setup.py to avoid path hacks. Install the wheel with pip and import the modules directly in a Databricks cluster.

Package your Python code as a wheel and install it on a Databricks cluster by uploading the wheel and referencing its path. Start cluster and import modules from the wheel.

Attach the wheel as a dependent library in the job task to install it on the job cluster, enabling native imports in notebook and Python tasks via the CLI bundle.

Configure a Python wheel task in Databricks to install a wheel on the cluster using the package name from setup.py and a main entry point.

Discover Delta Live Tables pipelines as a declarative ETL framework for batch and streaming, with schema inference, quality checks, and support for writing to schemas across bronze, silver, and gold.

Create the dlt pipeline in the ui, refactor code into a yaml bundle with catalog variables, and deploy to dev, test, and prod while validating bronze, silver, and gold notebooks.