Databricks Certified Data Engineer Professional -Preparation

Understand bronze ingestion patterns and compare singleplex (1-to-1) versus multiplex (many-to-one) mappings to a single bronze table, with streaming considerations and silver-layer filtering by topic.

Build a multiplex bronze table to drive three independent data pipelines by ingesting json files from cloud storage with autoloader, parsing timestamps, and partitioning by topic and year_month.

Learn streaming from a multiplexer bronze table to build an orders silver table by parsing JSON with from_json, casting kafka binary fields, and using a streaming temporary view.

Learn streaming deduplication in structured streaming by applying insert-only merge at the silver table, using watermarking to limit state and avoid duplicates across microbatches, while preserving bronze history.

explore slowly changing dimensions in the lakehouse, compare type 0, 1, and 2 SCDs, and implement type 2 with current, start date, and end date.

Implement a type 2 scd for books by merging updates via foreachBatch into the books silver table, marking old records with end_date and inserting new records with current status true.

Explore change data capture (CDC) and processing in Delta Lake, using merge into with rank functions to apply the most recent inserts, updates, and deletes from the source.

Process change data capture feeds to build the customers' silver table by ranking latest updates per customer, streaming with foreachBatch, performing merges, and enriching data via a country lookup.

Explore Delta Lake CDF, a change data feed that records row-level inserts, updates, and deletes in table_changes with pre- and post-images, and learn how to enable and query changes.

Enable delta change data feed on the customers table, then read changes from version 2 onward using the table changes function, exploring inserts, updates, and the underlying _change_data parquet metadata.

Propagate changes from CDF data by performing a stream-stream join to create the silver table customers_orders. Upsert ranked updates, filter inserts and update_postimage, and use watermarks to limit state.

Explore how stream-static join works in Spark Structured Streaming, contrasting streaming append-only tables with static Delta tables and their limitations.

Join a streaming orders dataset with the static current_books table to create a stream-static join on the silver books_sales table, using Delta Lake guarantees and batch-triggered appends.

Explore creating stored views and gold tables in Databricks, aggregating daily sales by country and author, with delta caching and five-minute intervals, plus late data handling with a ten-minute watermark.

Partition Delta Lake tables to optimize queries with year and month, using partition by columns, keeping partitions large enough, and applying Delta Lake optimize commands, deletes, and vacuum.

Explore delta lake partitioning on a bronze table by topic and year_month, inspect partition directories and delta log, and archive or delete old data with directory-level access control for PII.

Explore data file layout optimization in delta tables, using partitioning, z-order indexing, and liquid clustering to accelerate queries with data skipping.

Explore how Delta Lake uses a transaction log with JSON commits and Parquet checkpoints to speed state resolution, leverage file statistics for fast queries, and manage log retention.

Explore the delta lake transaction log, review per-file statistics, and see how the query optimizer uses them to prune data and speed counts. Learn about json and parquet checkpoints.

Discover how Databricks' auto optimize feature automatically tunes Delta Lake file sizes with optimized writes and auto compaction, producing 128 mb files to speed up merges.

Explore Python UDFs in Spark, register them for Python and SQL, and optimize with pandas UDFs using Apache Arrow for faster, vectorized operations on a books table.

Orchestrate a multi-task bookstore data pipeline with Databricks jobs, configure the land new data task and bronze, silver, and gold tasks, and set job clusters, retries, and dependencies.

Master advanced Databricks jobs configurations by scheduling with cron expressions, editing permissions, and managing ownership and run permissions to control who can view, run, or manage jobs.

Explore common causes of job failures and hands-on debugging, including widget not defined errors; learn how root task failure skips dependents while parallel tasks continue, and how to repair runs.

Demonstrate propagating deletes by processing delete requests for PII, updating downstream tables with foreachBatch, and concluding with vacuum commands to fully remove data.

Discover lakeflow declarative pipelines, a Spark ETL framework that simplifies building pipelines with Python or SQL syntax. Learn types—streaming tables, materialized views, and temporary views—and orchestration, checkpoints, and quality controls.

Refactor the bookstore ETL with lakeflow declarative pipelines, creating a streaming bronze table and a country lookup view, and set up the Bookstore Pro Pipeline.

Define data quality expectations in lakeflow declarative pipelines using SQL or Python with the expect function, and track violations in metrics, applying actions like warn, drop, or fail.

Implement data quality expectations in a lakeflow pipeline, creating orders_silver with valid_quantity, using @expect and decorators, handling violations via quarantine tables or is_quarantined flags.

Explore Auto CDC APIs in SQL and Python syntax, enabling out-of-sequence handling and SCD type 1 and 2 updates via Lakeflow declarative pipelines, avoiding hundreds of lines of code.

Leverage Auto CDC APIs to process CDC customer data and apply SCD type 2 updates to books, while creating materialized views for gold layer analytics in lakeflow declarative pipelines.

Explore the Databricks query profile to troubleshoot bottlenecks, review metrics like wall-clock and aggregated task time, top operators, and view the full query text in SQL editor and notebooks.

Schedule an ETL pipeline in a Databricks job, land new data, use an if/else condition to process or log, then list tables and iterate for per-table analysis.

Learn how databricks asset bundles automate deployment of databricks jobs and pipelines across dev, staging, and prod environments. Use bundle init, validate, deploy, and run to manage ci/cd integration.

Explore databricks asset bundles using a vscode workflow, create a bundle, configure the yaml, deploy a bookstore workflow, and run pipelines and jobs in development and production.

Master relative imports in Databricks notebooks by replacing the %run magic command with Python module imports, configuring sys.path, and importing from helpers and modules packages.

Explore how to use the Databricks REST API to authenticate with a bearer token, create and manage jobs, trigger runs, and monitor status via Postman.

Configure the Databricks CLI to manage clusters, DBFS, and secrets from the terminal; learn to upload files, view statuses, and read secrets with secret scopes.

Explore unity catalog concepts, including default workspace catalogs and schemas, privilege management, AI-generated comments, tags and policies, data quality monitoring, and billing usage analytics.

Create dynamic views in databricks to enforce column and row level access with acl, redact sensitive fields for non members using is_member(), and grant full access to admins_demo members.

Delta sharing is an open protocol for secure data sharing via a Delta sharing server and Unity Catalog, enabling Databricks-to-Databricks sharing and open sharing with time travel and read-only access.

Explore lakehouse federation to query external sources without data ingestion, enabling live access and ad hoc reporting with Unity Catalog, and use materialized views or ingestion for heavy queries.

Explore data pipeline testing on Databricks, covering data quality tests and standard tests. Apply unit, integration, and end-to-end tests with assertions and delta table constraints to validate data and code.

Configure admin and cluster-level permissions to control who can attach, restart, or manage clusters; then monitor events, driver logs, and Ganglia metrics to optimize performance.