Databricks Certified Data Engineer Associate - Bootcamp

Discover lakehouse, the fusion of data lake and data warehouse that powers Databricks, blending SQL capabilities with a scalable storage layer to meet volume, velocity, and variety challenges.

Delta Lake is an open table format on your data lake that enables SQL, ACID properties, time versioning, and partitioning through a delta log on parquet files.

Explore how Apache Spark powers Databricks as a distributed computing engine and how Databricks manages clusters, notebooks, Photon, Unity Catalog, and Delta Live Tables for data engineering.

Discover Databricks architecture, including the compute plane and control plane, how the ui talks to back-end services to create clusters, and the differences between serverless and classic compute.

Learn how the medallion architecture uses three layers: bronze layer, gold layer, and representation layer, to transform raw data into a dimensional data model with fact and dimension tables.

Discover Databricks Free Edition, a fully free, no credit card platform that replaces Community Edition, offering production-ready workspace to practice Databricks features and PySpark without cloud accounts.

Explore Databricks overview, including free edition features, workspace organization, compute options, workflows, and SQL tooling, to master data engineering with lake flow declarative pipelines.

Refer to notebooks in Databricks using the dot dbc archive, import into a Databricks Reference folder in your workspace, and access the resources for .py and .sql files.

Explore Databricks compute options, including all-purpose, job, pools, SQL warehouse, and serverless, along with Unity Catalog concepts for development and production SQL workloads.

Explore how Unity Catalog provides centralized data governance across Databricks workspaces. Enforce policies, audit usage, and enable data discovery and lineage through tagging and documentation.

Explore how Unity Catalog centralizes the metastore and access management, enabling cross-workspace sharing of catalogs and schemas via a single Unity Metastore.

Compare managed and external tables in Databricks, detailing metadata in the metastore, data in storage accounts, and differing drop behaviors.

Create a bronze schema in Databricks using the UI, then create a managed table from an uploaded sales data CSV, which converts to delta data stored in Databricks internal location.

Learn how Unity Catalog volumes provide a governance layer over non-tabular data, enabling volume-based querying and registration under catalog and schema, with managed volumes and hierarchical folders.

Create and manage volumes in databricks, choosing between managed and external volumes, organizing bronze layer folders, and loading data via volume paths; explore notebooks and utilities.

Explore dbutils, the Databricks os module, to manage files, folders, secrets, and widgets with fs operations like ls, mkdir, put, cp, and rm in notebooks.

Master PySpark transformations for bronze to silver data processing in Databricks, including CSV reading with infer schema, domain extraction, aggregations, and upserts into Delta silver tables.

Explore Spark SQL as the SQL engine in Apache Spark, enabling SQL queries within PySpark and transforming data frames with SQL statements, including temporary views for notebooks.

Explore how Databricks files enable collaboration on the Databricks data intelligence platform, organizing bronze and silver folders and reading or writing text, JSON, and CSV files with Python and Spark.

Explore Lakehouse federation in Databricks by connecting to external data sources like SQL databases without pulling data, and test architectures with a POC to optimize compute costs.

Learn to set up a lakehouse federation federated catalog and read-only foreign tables in Databricks, enabling ad hoc queries on external data without migration for quick proofs of concept.

Discover Lake Flow Connect, a production-grade approach to ingesting data from sources into your native cloud with Databricks, aligning with Lake House Federation concepts.

Learn why Lake Flow Connect provides data ownership, low latency, and Delta Lake optimization when ingesting data into your data lake connected to Databricks.

Upload local CSV or JSON files with lake flow connect to auto-create delta format tables or volumes in one click, enabling no-code data ingestion from your Databricks workspace.

Explore the standard connector for Lakeflow Connect at Databricks, linking cloud storage like Azure Data Lake Storage, Autoloader, and external locations for batch and streaming workloads.

Explore the Lakeflow Connect manage connector, reducing ingestion overhead with a low/no-code UI, offering five native connectors and 100+ through Fivetran for easy data ingestion.

Explore Spark Streaming and Spark Structured Streaming for real-time data ingestion, processing, and loading from continuous sources, and learn how Databricks auto loader simplifies heavy maintenance.

Master the autoloader architecture in Databricks to incrementally load data from cloud storage with Spark structured streaming and ignore already processed files for streaming pipelines.

Explore how autoloader ensures idempotent, exactly-once processing using rocksdb for file metadata, while schema inference and evolution manage new columns with checkpoint tracking.

Explore setting up databricks autoloader for streaming csv data, configuring cloud files with schema and checkpoint locations, enabling schema evolution rescue and incremental delta writes.

Explore idempotency with autoloader by uploading files to a volume, observing that only new data is processed, ensuring exactly-once semantics and rescue data handling for evolving schemas.

Explore how autoloader handles schema evolution by uploading files with different structures, rescuing data using the latest schema from the schema location, and fetching newly added columns.

Add new columns with merge schema true in autoloader to automatically extend destination schemas, compare with rescue data, and observe behavior across multiple files in bronze volumes.

Explore the copy into command to incrementally load data from file paths into a delta table, ensuring idempotent, SQL-first loading with a table destination.

Demonstrate how copy into loads parquet files with built-in schema into Delta Lake, enabling schema evolution and idempotent updates for bronze tables.

discover databricks sql as a self-contained data warehousing layer on your data lake, offering ANSI sql compatibility, a built-in sql editor, dashboards, and collaborative visualization tools.

Discover how the SQL warehouse provides dedicated compute for SQL workloads in Databricks, featuring serverless auto scaling, concurrency management, and easy connection details for the SQL editor.

Explore the Databricks SQL editor UI to write queries, explore data, and build visualizations, while saving, sharing, and managing SQL queries and files across catalogs and schemas.

Learn how to use Databricks SQL to build parameterized queries with dbutils widgets, including value and object parameters, colon syntax, and the identifier function for dynamic column selection.

Explore query snippets to reuse SQL code, creating static and dynamic snippets with placeholders in the Databricks query editor, and apply them to common where clauses like category toys.

Use common table expressions to build ranked product insights from sales and products data. Parameterize top N results using with clauses and dense_rank, and explore sharing and scheduling.

Learn to schedule queries in Databricks by using a calendar-like UI, set cron-based timings, time zones, and a SQL warehouse, and manage jobs and pipelines from creation to deletion.

Explore the monitor hub to view and filter query profiles in Databricks SQL, analyze memory peaks, rows processed, and transformation steps from the query profile for effective monitoring.

Learn how query caching speeds dashboards by reusing cached results when the data and user and exact query are unchanged, avoiding recomputation.

Explore Databricks Genie, an AI agent that turns natural language into SQL over your catalogs and tables, enabling creation, training, and running of queries with joins and viewable code.

Explore Databricks SQL dashboards to quickly create and customize visualizations, integrate data sources, and publish, share, or schedule refreshes, using AI-assisted chart generation for data analysts and data engineers.

Master lake flow jobs, the ETL pipelines in Databricks, to build workflows, orchestration, and ML pipelines for training and deploying models using a no-code visual UI with tasks and dags.

Create your first lake flow job in Databricks by orchestrating a Python notebook and a SQL file, using a serverless compute canvas to run tasks in sequence.

Explore conditional logic with if and else in Databricks jobs, using dynamic content, job start time, and boolean checks to control task dependencies and notebook or SQL execution.

Learn how to apply a for-each loop in Databricks pipelines to run an activity over an array, control concurrency, and execute SQL or notebook tasks within a loop.

Master dynamic value references to pass data between SQL and Python notebook tasks, using task values, db utils, and get task values to build real-time pipelines.

Learn to manage a free Databricks setup and prevent quota limit exceed errors by cleaning alerts, dashboards, and jobs, and by running one focused workflow at a time.

Explore how to set and get values in Databricks jobs by building dynamic notebooks and sql scripts that pass parameters and retrieve results with dbutils task values.

Convert SQL rows into a list for for-each activities using the rows function, producing dictionaries of column names and values, with eval converting text to a list in Databricks.

Learn how the first row output returns a single dictionary of key-value pairs. Compare this to the rows output and demonstrate SQL row and SQL column retrieval in a notebook.

Apply for each over SQL rows in Databricks to run an inner SQL script per row, using input dot product ID as a parameter.

Master dynamic data ingestion by passing large arrays into notebooks using dbutils widgets and foreach loops to copy data from bronze to raw folders with folder name parameters.

Replace the notebook lookup with a SQL mapping table in Databricks bronze, two columns (parent folder name, folder name), producing arrays of dictionaries for each activity.

Use scheduling and triggers in Databricks to automate pipelines through scheduled runs, file arrival (storage events) triggers, or continuous for streaming.

Choose and swap compute types in the databricks compute tab to attach different compute to your jobs, including all purpose compute and job compute available with a paid subscription.

Learn how to monitor Databricks jobs and pipelines using the monitoring tab, view job runs, timelines, and lineage, and use parameters and repair options to ensure production readiness.

Configure Databricks job notifications to receive immediate failure alerts via email or system destinations like Microsoft Teams and Slack, with Lake Flow Declarative Pipelines.