Databricks Data Engineering with AWS

Explore data engineering fundamentals and how Databricks and Apache Spark solve data problems. Learn the lake house medallion architecture with bronze, silver, and gold layers, and batch and streaming ingestion.

Databricks extends Apache Spark with cloud native lake house platform and photon query engine, integrating Delta Lake, Unity catalog, notebooks, and cluster management across Azure, AWS, and GCP.

Discover how the Databricks platform architecture delivers a multi-tenant SaaS across aws, azure, and gcp with a split control and serverless compute plane, and optional cross-account data storage.

Create a Databricks free edition account with email verification, access a free serverless workspace, set your account name and country, complete the puzzle, and customize the workspace and editor themes.

Learn how to set up a Databricks premium account in the AWS cloud, and create serverless and classic workspaces. Navigate AWS marketplace contracts and cost-saving cleanup.

Learn how to view AWS billing in the billing and cost management console, configure a budget, and set email alerts to monitor net costs against credits and the free tier.

Learn how to set up a Databricks account on AWS marketplace, create a serverless workspace, link your subscription, and understand the 14-day credit window and pay-as-you-go compute.

Explore notebook magic commands for language switching (percent sql, scala, md), file system operations (%fs), and modular workflow using %run to connect multiple notebooks in Databricks.

Learn to debug notebooks in the Databricks environment by enabling the Python notebook interactive debugger, attaching a cluster, and using breakpoints and df.show to inspect data.

Learn three delta table creation approaches in Databricks—simple create table, create table as select, and Spark dataframe write—and distinguish managed from external tables and their data ownership.

Master delta lake time travel to query historical versions via version as of or time stamp as of, using delta transaction log for auditing, debugging, and rollback, noting vacuum limits.

Explore Unity Catalog architecture in Databricks, from a central metastore to catalogs and schemas, with external locations and volumes, enabling cross-workspace visibility and unified permissions.

Learn to configure Databricks unity catalog with two catalogs tied to separate S3 buckets via external locations and storage credentials, and assign workspace access to prevent cross-team visibility.

Implement Databricks user management by creating users and groups, introducing a service principal, and granting group-based access to dev, uat, and prod workspaces, with catalog visibility considerations.

Demonstrate implementing unity catalog permissions at the catalog and schema levels, granting use catalog and create schema to data engineering and uat groups while denying metastore and external location access.

Explore the medallion architecture for data engineering, with bronze, silver, and gold layers that separate raw data, business logic, and consumption, preventing data swamp and enabling end-to-end implementation.

Build the gold layer as a pre-aggregated, read-optimized table for a specific consumer, choosing materialized tables over views to report daily revenue by customer tier from completed orders.

Ingest data to the bronze layer from a SaaS app using databricks managed connectors and a no-code pipeline, pulling GitHub commits and issues.

Implement a query-based connector for incremental ingestion from a relational database in Databricks, configuring connections, pipelines, and a cursor column to capture inserts and updates (deletes are difficult to capture).

Discover how Lakeflow SDP transforms data engineering by declaratively defining end-to-end pipelines that manage the bronze, silver, and gold stages with CDC and data quality, reducing boilerplate.

Explore pipelines, flows, and destinations in SDP, including append, auto CDC, materialized view, and update flows, and learn the Python DP API for building them.

Extend the pipeline by adding silver layer auto cdc with scd type 2 for orders and customers. Use explicit syntax to create silver streaming tables and auto cdc flows.

Create and orchestrate a lakeflow job to run the medallion pipeline daily, with tasks, a pipeline task, and a data quality notebook, including schedule, triggers, dependencies, and email notifications.

Learn to parameterize Lakeflow jobs in Databricks with AWS by using job and task parameters, passing run date via notebook widgets, and leveraging dynamic value references and task values.

Learn how to pass task values between jobs, receive job parameters in a Python script via command line arguments, and use repair runs to restart failed executions.

Learn to design a multi-dag lakeflow pipeline with control flow, retries, and backfill, including DQ gold checks, alerting, threshold-based reporting, and low data warnings.