dbt (Data Build Tool) : dbt for Data Engineers

Explore the data warehouse concept as a time-variant, non-volatile, integrated repository for historical data from heterogeneous sources; learn etl workflows, staging, and how business intelligence tools analyze denormalized summary data.

Explore the advantages of a data warehouse, including historical intelligence, faster queries from denormalized data, centralized access, and return on investment, while noting limitations like maintenance costs and ownership concerns.

Master ETL in a data warehouse, from mediation system and extraction methods (logical vs physical; full vs incremental; online vs offline) to transformations and loading strategies (initial, incremental, full refresh).

Compare ETL and ELT: ETL extracts to a separate staging server, transforms there, and loads to data warehouse; ELT loads first and transforms on the target database for better performance.

Learn how to use the ref function in dbt to reference one model from another, join store_2023 and store_sales_2023, and build a final aggregated view or table.

Build a python model in dbt by defining a python module with def, using Snow Park library, and using dbt ref to join frames and aggregate by name and country.

Learn how dbt sources work by defining a source in a yaml file, referencing the source function, and migrating tables between schemas while tracking lineage and executing models.

Explore how source freshness in dbt tests whether source tables are updated within a set window, demonstrated with an employee table and a loaded_date field.

Explore seeds in dbt, using CSV files in the seed directory, loading them into the data warehouse with dbt seed, and referencing them in models via ref and YAML.

Materialization in dbt defines how models are stored as tables, views, ephemeral, or incremental. Views auto-update; tables require redeployments, while ephemeral and incremental offer lightweight or targeted updates.

Learn how the incremental model processes only new or changed records from the source table, using Snowflake's default merge strategy and the delete plus insert option.

Implement type 2 in db2 by using snapshot-based history with a timestamp method, leveraging the updated_at field to detect changes and maintain past employee records.

Explore implementing SCD type 2 with dbt's check strategy by configuring a snapshot on employee name and state to detect changes.

Explore using Jinja in dbt to replace multiple case statements with a for loop, building region-wise totals and compiling transformed SQL without execution.

Explains macros in dbt as function-like reusable code and demonstrates building a dept_salary macro used in a top salary model to extract department top salaries.

Learn to pass arguments to a dbt macro by building an employee macro and calling it from a model, with salary and department id filters and compiled SQL in Snowflake.

Learn to use dbt hooks to run sql before or after models, including pre-hook and post-hook, and on run start and end, for tasks like altering tables.

Document your dbt models with doc blocks to record data types, keys, and metadata using schema.yaml. Generate and view dbt docs to examine model details, version history, tests, and lineage.