Databricks Data Engineering with AWS

Explore data engineering fundamentals and how Databricks and Apache Spark solve data problems. Learn the lake house medallion architecture with bronze, silver, and gold layers, and batch and streaming ingestion.

Databricks extends Apache Spark with cloud native lake house platform and photon query engine, integrating Delta Lake, Unity catalog, notebooks, and cluster management across Azure, AWS, and GCP.

Discover how the Databricks platform architecture delivers a multi-tenant SaaS across aws, azure, and gcp with a split control and serverless compute plane, and optional cross-account data storage.

Create a Databricks free edition account with email verification, access a free serverless workspace, set your account name and country, complete the puzzle, and customize the workspace and editor themes.

Learn how to set up a Databricks premium account in the AWS cloud, and create serverless and classic workspaces. Navigate AWS marketplace contracts and cost-saving cleanup.

Create an AWS account, verify email, and secure the root user with MFA, then set up an IAM user for daily work and choose a learning-friendly region.

Create an AWS IAM user for day-to-day work by logging in as the root user, enabling MFA, adding the user to an admin group, and configuring permissions and region settings.

Learn how to view AWS billing in the billing and cost management console, configure a budget, and set email alerts to monitor net costs against credits and the free tier.

Learn how to set up a Databricks account on AWS marketplace, create a serverless workspace, link your subscription, and understand the 14-day credit window and pay-as-you-go compute.

Create a Databricks classic workspace in AWS by linking your account in the marketplace, configuring an S3 workspace bucket and IAM roles, then launch in US East 1.

Explore how the Databricks workspace serves as an integrated development environment. Compare serverless and classic workspaces, review compute options, SQL warehouse, and unity catalog.

Explore notebook magic commands for language switching (percent sql, scala, md), file system operations (%fs), and modular workflow using %run to connect multiple notebooks in Databricks.

Learn to use Databricks utils and widgets to automate file system tasks, manage notebooks, and parameterize workflows with dbutils.fs, dbutils.notebook, and widget utilities.

Learn to debug notebooks in the Databricks environment by enabling the Python notebook interactive debugger, attaching a cluster, and using breakpoints and df.show to inspect data.

Explore how Databricks workspace files and git folders store notebooks and code, compare their advantages, and learn when to use workspace files versus git folders for development with source control.

Identify Databricks compute options, including serverless and classic compute, with standard and dedicated clusters; learn auto scaling, policies, and when to use each for scalable, cost efficient data engineering.

Understand Delta Lake as a storage layer that adds a transaction log to Parquet data, enabling acid transactions, time travel, and data skipping, with Unity Catalog governance on Databricks.

Learn three delta table creation approaches in Databricks—simple create table, create table as select, and Spark dataframe write—and distinguish managed from external tables and their data ownership.

Master delta lake time travel to query historical versions via version as of or time stamp as of, using delta transaction log for auditing, debugging, and rollback, noting vacuum limits.

Explore Delta Lake restore and rollback strategies, using restore by version or timestamp, time travel with merge for selective rollback, and shallow or deep clones for testing.

Understand how the Delta Lake merge statement performs atomic upserts by updating, inserting, and deleting in one transaction. Learn typical patterns, scenarios, and performance considerations.

Learn how delta lake handles delete and update with deletion vectors and automatic optimization, and implement idempotent writes using transaction application id and version to achieve exactly-once semantics.

Delta Lake enforces schema by default, rejecting writes with unknown columns or mismatched types, and offers merge schema, alter table, and generated columns for controlled evolution.

Discover Delta Lake type widening to upgrade column types within the same family and the variant data type for flexible JSON handling, including parse JSON and access using colon operator.

Define not null and check constraints in delta tables to enforce value-level rules, stored as table properties in the delta transaction log, and use identity columns to create surrogate keys.

Explore Unity Catalog architecture in Databricks, from a central metastore to catalogs and schemas, with external locations and volumes, enabling cross-workspace visibility and unified permissions.

Create a single Unity Catalog metastore and three Databricks workspaces, then connect all workspaces to that metastore to enable unified access across development, testing, and production.

Learn to configure Databricks unity catalog with two catalogs tied to separate S3 buckets via external locations and storage credentials, and assign workspace access to prevent cross-team visibility.

Learn how Databricks Unity Catalog manages identity with users, groups, and service principals, and how account and workspace admins control access and ownership across dev, uat, and prod workspaces.

Implement Databricks user management by creating users and groups, introducing a service principal, and granting group-based access to dev, uat, and prod workspaces, with catalog visibility considerations.

Apply Unity Catalog permissions from the metastore to catalogs and schemas, showing the grant flow from account admins or metastore admins to teams for using catalogs and creating schemas.

Demonstrate implementing unity catalog permissions at the catalog and schema levels, granting use catalog and create schema to data engineering and uat groups while denying metastore and external location access.

Explore the medallion architecture for data engineering, with bronze, silver, and gold layers that separate raw data, business logic, and consumption, preventing data swamp and enabling end-to-end implementation.

Preserve all incoming data in the bronze layer of the medallion architecture, adding ingestion metadata and loading as strings. Simulate inline CSV sources and show duplicates allowed for bronze behavior.

Designing the silver layer enforces contracts by deduplicating, typing, and validating data from the bronze layer, enabling reliable downstream gold analyses and explorations.

Build the gold layer as a pre-aggregated, read-optimized table for a specific consumer, choosing materialized tables over views to report daily revenue by customer tier from completed orders.

Master medallion architecture by deciding when to add bronze, silver, and gold layers, what goes in silver versus gold, and how governance with Unity Catalog prevents sprawl.

Ingest data to the bronze layer from a SaaS app using databricks managed connectors and a no-code pipeline, pulling GitHub commits and issues.

Implement a query-based connector for incremental ingestion from a relational database in Databricks, configuring connections, pipelines, and a cursor column to capture inserts and updates (deletes are difficult to capture).

Learn how to ingest data into the bronze layer with autoloader, a standard connector for cloud storage files, enabling incremental ingestion, schema evolution, and metadata tracking for orders and customers.

Discover how Lakeflow SDP transforms data engineering by declaratively defining end-to-end pipelines that manage the bronze, silver, and gold stages with CDC and data quality, reducing boilerplate.

Explore pipelines, flows, and destinations in SDP, including append, auto CDC, materialized view, and update flows, and learn the Python DP API for building them.

Extend the pipeline by adding silver layer auto cdc with scd type 2 for orders and customers. Use explicit syntax to create silver streaming tables and auto cdc flows.

Implement the gold layer with a metallized view that reads silver layer data, joins orders and customers, and outputs total revenue, order count, and average order value.

Understand why Lakeflow jobs provide reliable orchestration for production pipelines, with jobs, tasks, and triggers driving notebook, pipeline, and Python script tasks.

Create and orchestrate a lakeflow job to run the medallion pipeline daily, with tasks, a pipeline task, and a data quality notebook, including schedule, triggers, dependencies, and email notifications.

Learn to parameterize Lakeflow jobs in Databricks with AWS by using job and task parameters, passing run date via notebook widgets, and leveraging dynamic value references and task values.

Learn how to pass task values between jobs, receive job parameters in a Python script via command line arguments, and use repair runs to restart failed executions.

Learn to design a multi-dag lakeflow pipeline with control flow, retries, and backfill, including DQ gold checks, alerting, threshold-based reporting, and low data warnings.

Select production task type, use pipeline task for sdp updates and notebook or python script tasks for data quality, and define job parameters at the job level with repair runs.