dbt on Databricks

Configure a Databricks workspace on Azure with Unity Catalog, create an A2X small serverless SQL warehouse set to auto stop after ten minutes, and set up a dbt project catalog.

Load and structure a retail data set in a Databricks lakehouse by creating a landing schema, importing four parquet tables, and linking orders, products, reviews, and users for DBT processing.

Explore DBT cloud, a managed service on DBT core with a browser-based IDE, scheduling, and version control. Sign up, navigate the UI, and manage plans, projects, and environments.

Set up a dbt cloud project for databricks, configure a connection with hostname and http path, enable unity catalog, generate a development token, and initialize the repository with initial commit.

Dbt translates SQL models into executable queries for your data lake, house, or warehouse, revealing the core components like models, compilation, lineage, tests, seeds, macros, and project YAML.

Learn to manage a shared dbt project with git in the dbt cloud IDE, creating feature branches, committing changes, and merging into main while handling reverts and file changes.

Explore dbt on Databricks using medallion architecture with bronze, silver, and gold layers, ingesting landing data into bronze and building silver and gold tables for orders, daily sales, and ratings.

Explore the course wiki for dbt on Databricks, accessing sql and yaml code snippets, and following along with bronze_orders sql and dbt project yaml configurations.

Create a bronze_orders model in dbt on Databricks to materialize landing.orders as a bronze table or view in the development schema, and switch materializations via the project YAML.

Learn how to clean compiled artifacts in dbt on Databricks by using the dbt clean command to remove target folders and recompile on the next dbt run.

Configure landing sources with yaml, use the source function to link to bronze models, and run dbt to materialize bronze orders, products, reviews, and users with lineage in Databricks.

Learn to check data freshness in dbt using a freshness block in yaml, applied at source or table level, with warn and error thresholds based on a timestamp field.

Use dbt to build the silver layer by referencing bronze tables to create silver orders, products, and users, performing light cleaning and deriving order date, order amount, birth year, and sales channel.

Model the gold layer by building gold daily sales and gold daily average rating tables via joins of silver orders, products, users, and bronze reviews, then run dbt and commit.

Organize your dbt on Databricks project with a clear folder structure and naming conventions, aligning staging bronze, silver, and gold layers to improve navigation and collaboration.

Learn to define properties and configurations in dbt on Databricks. Use a properties YAML to document models and columns, and manage materialization with project YAML, properties YAML, or model configs.

Learn to configure custom bronze, silver, and gold schemas in dbt on Databricks by using project YAML, properties YAML, and model-level config or macros to control schema names.

Master dbt node selection to run specific models and dependencies using the select, plus, and exclude options, and tag models containing personally identifiable information with config block to control execution.

Explore generic and singular tests in dbt on databricks to validate data integrity, apply out-of-the-box checks like unique and not null, and craft custom sql tests.

Explore how dbt on Databricks uses tests and data tests in YAML configurations, and how backward compatibility lets you choose either naming convention.

Configure generic tests for orders, products, reviews, and users in dbt on databricks, focusing on not null and unique checks, then run tests and adjust severity to warn.

Add generic tests to models with a properties YAML, configuring accepted values. Ensure referential integrity by linking bronze_products id to bronze_orders product_id with a relationships test and dbt test.

Explore singular sql tests in dbt on Databricks, placing custom tests in the tests directory to enforce business rules with zero-row results and configure severity for passes and failures.

dbt build combines run and test steps into one command, automatically testing each DAG layer (sources, bronze, silver, gold) before materializing downstream models, with select and exclude options.

Load static data by uploading CSV seeds into the data lake house as tables with dbt seed, and configure schema via YAML or properties for reference data and lookups.

Explore dbt analyses on databricks, where SQL files in analyses directory compile for ad hoc queries and data exploration without materializing into models, using the reference function for environment-agnostic selection.

Explore Jinja syntax for dynamic SQL in dbt on Databricks, using double curly braces for expressions, percent braces for control structures, and macros and variables for reusable, adaptable code.

Master setting and outputting variables in Jinja with set blocks and inline syntax, manage whitespace with minus operators, and access list elements by index.

Define a temperature variable and apply an if conditional statement to return 'it is hot', 'it is not hot', or 'it is freezing', using if, elif, and else.

Learn how to use for loops in ginger to iterate over lists and generate dynamic SQL. Build a select statement with loop.last and if conditions to avoid trailing commas.

Learn how to use built-in dbt functions on Databricks, including ref and source for models and data, and modules like date time and adapter to get columns in relation.

Explore how macros in dbt on databricks use Jinja to create reusable SQL logic with arguments and defaults, demonstrated by current_timestamp and multiply_columns, and applied to models for streamlined code.

Explore how dbt packages extend projects by importing reusable components from the dbt hub, featuring the dbt utils date spine macro and installation via packages.yaml or git.

Explore dbt materializations on Databricks, including views, tables, incremental, ephemeral, and snapshots, with pros, cons, and guidance on choosing the right approach for your models.

Recap how to configure dbt materializations for bronze, silver, and gold layers in Databricks, switching between tables and views and understanding precedence of config blocks, properties YAML, and project YAML.

Configure dbt models as incremental to append new rows, demonstrated with orders incremental and bronze orders, and use the maximum createdat timestamp to avoid duplicates; explore the full refresh option.

Learn how to upsert records from source to target using a unique key to replace duplicate rows, enabling incremental materialization in dbt on Databricks.

Use common table expressions with the with clause to create temporary result sets and reference CTEs in the main query, demonstrated in a dbt project for gold average rating daily.

Learn how ephemeral models run in memory for transformations, letting dbt embed them in queries, and how to switch silver models to ephemeral, perform drops, and balance performance vs storage.

Discover how to implement slowly changing dimensions type two with dbt snapshots, using timestamp or check strategies to track product price changes over time in the bronze layer.

Configure Databricks-specific table options when materializing a model as a table, including an alternate file format, custom storage locations, and partitioning by selected columns.

Recap macros and packages in dbt on Databricks, review generic tests for sources and models, and explore making singular tests more reusable with dbt utils and dbt expectations.

Learn how to run targeted dbt tests on models and sources using the select argument, test types, and dot notation to include upstream or downstream dependencies.

Explore advanced test configurations in dbt on Databricks, including error and warn thresholds, where clauses for active records, uniqueness tests on snapshots, and storing failures.

Create and apply custom generic tests to models and columns in dbt, using a reusable macro or generic folder, enforcing non-negative unit price and quantity on the landing orders source.

Use the dbt_utils package to apply the expression is true macro for reusable tests, validating non-negative quantity and unit price in a dbt on Databricks project.

learn to validate data in your dbt workflow with dbt expectations, using predefined tests for table shape, column lists matching a set, missing values, unique values, types, and distributions.

Explore how dbt on Databricks uses unit testing to validate individual data model logic before building models, using given inputs, mock data, and expect blocks for targeted SQL checks.

Explore unit testing in a dbt on Databricks project by building and testing silver and gold models with YAML tests, CSV data, and macros like multiply columns and round.

Merge your dbt workflows to production by consolidating feature branches into main, while switching gold layer models to views and performing a full refresh of bronze orders.

Understand dbt cloud deployment environments, using a per-developer development environment with individual credentials and bronze, silver, and gold schemas, and configure general, staging, and production deployment environments for live pipelines.

Create a production deployment for a dbt on Databricks project, configure a prod environment with main-branch execution, catalog production overrides, and token-based credentials to run jobs.

Learn to create and run your first production dbt deploy job on databricks, configure settings, enable dbt build and docs, and verify bronze to gold schemas.

Dbt on Databricks creates a landing schema in the prod catalog, copies the landing schema from the old catalog, and creates tables to prepare data ingestion.

Leverage target variables to adapt dbt configurations for development and production. Use conditional jinja logic in landing sources to switch catalogs by target.name, enabling prod catalogs in production.

Configure dbt on Databricks with environment variables, leveraging jinja env underscore var to set catalogs per environment in landing sources yaml and manage development and production deployments.

Add descriptions to dbt on databricks by using YAML description keys for sources and tables, including folded or literal styles, and link markdown doc blocks via the doc function.

Learn how pre hooks, post hooks, and on run start/end hooks automate customizable sql statements around dbt models and seeds or snapshots, enabling prerequisites, cleanup, logging, and auditing.