Databricks: Spark Declarative Pipelines & Lakeflow Designer

Understand how Delta Lake works and why features like ACID transactions, schema enforcement, and time travel are critical for reliable pipelines.

Learn how to organize data using catalogs, schemas, and tables, and understand governance, access control, and data lineage in Databricks.

Learn the core idea behind Spark Declarative Pipelines and how the Bronze → Silver → Gold structure organizes your data workflows.

Ingest raw data into a Bronze table using declarative pipelines while preserving the original structure for traceability.

Clean and transform the data by fixing data types, standardizing fields, handling null values, and preparing it for downstream use.

Create analytics-ready tables with business metrics like revenue and customer activity using aggregations on the refined data.

Run and schedule your declarative pipeline in Lakeflow Designer and verify how data flows through Bronze, Silver, and Gold.

Start building a new pipeline visually in Lakeflow Designer using a canvas-based approach and guided configuration.

Extend the visual pipeline by adding transformations, combining datasets, and generating the underlying SQL logic.

Understand how streaming works in declarative pipelines and the difference between triggered and continuous execution modes.

Set up a streaming data source by connecting Databricks to an AWS message queue and configuring ingestion.

Build a streaming pipeline that ingests real-time data into Bronze and processes it into Silver with basic transformations.

Recap the key concepts from the course and understand how to apply declarative pipelines in real-world projects.