DP-750: Microsoft Azure Databricks Data Engineer Associate

Explore how lakehouse unifies data lake storage with an open table format and delta lake to enable SQL-like queries, transactional logging, and Spark-ready analytics on Databricks.

Configure external locations and credentials in unity catalog to securely access data in your data lake from Azure Databricks, using storage credentials and metastore permissions.

Compare external and managed tables on Databricks, explaining metadata in the metastore and data in cloud storage, and how dropping a managed table deletes data while external tables preserve it.

Explore external versus managed catalogs and schemas, how data links and the unity metastore determine storage locations, and when to use managed versus external configurations.

Explore how Databricks compute powers the cluster to run data engineering, data science, and analytics tasks, and understand CPU, RAM, and disk storage. Compare classic, SQL warehouse, and serverless compute.

Discover SQL warehouse compute in Azure Databricks, a SQL workload compute to query data. Compare classic and pro options with photon and predictive IO for analytics and reporting workloads.

Learn how batch ingestion works as the data ingestion design, with data loaded in batches into data lakes like ADLS Gen2, explaining why 80 percent of solutions rely on it.

Learn to use copy into for incremental ingestion from data lake to a table, with automatic handling of new files and flexible schema that adds columns.

Ingest data from public APIs using Python requests, convert JSON into a Spark data frame, and store in Delta format with append, overwrite, or ignore options, optionally creating a table.

Explore how autoloader handles schema evolution with Delta Lake, adding new columns by default, merging schemas with merge schema true, and rescuing data to populate fields without breaking downstream data.

Demonstrates upserts with update commands on an orders table, updating customer names and the updated_at timestamp while covering date time offset casting and how changes surface in Databricks pipelines.

Create a JDBC connection in Azure Databricks by building the JDBC URL with host, port, and database, set credentials, then ingest data from SQL Server into a delta table.

Demonstrate building an Azure Data Factory data lake ingest pipeline, using copy data activity to move JSON files from source to a destination in Azure Data Lake Storage Gen2.

Transform diverse data formats from a data lake into a gold layer via bronze and silver, delivering an OBD one big table for stakeholders using Spark SQL.

Transform bronze tables to the silver layer in Azure Databricks and learn to handle nulls and missing values with dropna and fillna, using subset for targeted cleaning.

Transform complex date strings in PySpark by converting multiple formats to date and timestamp, using coalesce and Unix timestamps, to fuel silver and gold layers in Databricks.

Understand spark sql file connectors in Databricks to read data from data lakes or volumes. Use parquet, csv, json, or delta formats to run ad hoc analyses with spark sql.

Databricks Unity Catalog functions enable governance and reuse of scalar UDFs stored in the catalog, with SQL or Python implementations applied across data, including handling nulls.

Explore dbutils, the Databricks OS module, and its fs commands like ls, cp, mkdirs, and rm for dbfs and data lake paths, plus widgets, secrets, notebooks, and jobs.

Explore Lakeflow declarative pipelines, master components and real-world use cases, build a star schema, and work with streaming sources like Event Hubs.

Create a delta table as a dual batch-and-stream source, then build an LDP pipeline in Azure Databricks using the new code editor, managing transformations, folders, and schemas.

Learn to build a streaming table in Databricks using PySpark and declarative dynamic pipelines, defining ingestion, transformations, and streaming write with checkpoint management.

Discover how to create streaming views or normal views in Databricks, using read stream vs spark.read.table, with function name naming and auto dependency tracking for real-time data pipelines.

Explore how materialized views compute by maintaining a persisted state and using incremental, append-only streaming data; they cache grouped results and refresh at intervals.

Build a real-world end-to-end data pipeline with declarative pipelines, streaming bronze to silver to gold layers, including stream-to-static joins and a star schema.

Build silver layer from bronze trips in Databricks, edit the pipeline, apply spark transformations, uppercase trip status, and add silver processed at timestamp using a streaming read from bronze trips.

Create the OBD gold table by streaming trips and joining with static tables using stream-to-static joins and left joins. The lecture introduces watermarks in Spark structured streaming.

Explore stream-to-static joins in Spark Structured Streaming with watermarking, using a 10-minute watermark to drop late records and join micro-batches to snapshot tables for star schema design.

Create a fact table by selecting IDs—trip, customer, driver, vehicle, and payment—to form the grain and keep it as an append-only table, with optional surrogate keys.

Explore append flow in Azure Databricks, merging data from data lake and delta tables into a single streaming table via an append flow that dumps data one by one.

Learn to parameterize Databricks data pipelines by creating key-value parameters in configuration, retrieving them via spark.conf.get, and using them to drive source tables and loops.

Learn how to schedule LDP pipelines using Databricks jobs or the pipeline's built-in scheduling, including cron-based timing, time zone alignment, and optional notifications.

Learn how to monitor and debug data pipelines with LDP in Databricks, using out-of-the-box monitoring, runs, logs, and performance metrics to quickly identify failures.

Explore how Azure Event Hubs provide Kafka-compatible storage layer for streaming events, enabling Databricks ingestion and real-time analytics via topics, consumer groups, and policies.

Create a streaming table on Azure event hubs in the Databricks LDP pipeline by configuring event hub namespace, topic, and a spark.readstream kafka stream using spark.conf.get for the connection string.

Configure and persist your Databricks event logs by exporting pipeline events to Unity Catalog as a managed delta table for dashboards and alerts.

Explore triggered vs continuous pipeline modes in Databricks, learn scheduling, near real-time processing, and how to configure trigger intervals and guardrails for continuous pipelines.

Learn about the Databricks SQL Warehouse, a simple, easy-to-use area you explore to understand what it is and how to use it in daily Databricks activities.

Explore how Databricks SQL functions as a cloud data warehouse built on a lakehouse, enabling SQL workloads with cost-effective, high performance without moving data.

Explore parameterized queries in Databricks SQL, using colon parameters to create reusable inputs, share and rename queries, and organize them into folders for automated ad hoc analysis.

Explore query caching by saving query state to avoid recomputing when data or the query hasn't changed. Outline ui and result caches in Databricks, with per user and cluster scopes.

Compare SQL query and SQL file in Databricks, demonstrating SQL blocks in notebooks, file creation with .sql, and why queries offer better performance for ad hoc analytics.

Explore how to create and configure a Databricks job, add tasks (notebook, Python script, SQL) with scheduling, permissions, alerts, and performance settings, then plan PySpark notebook scheduling.

Explore task values in Azure Databricks workflows, detailing stateless versus stateful orchestration and how to set and get values across notebooks using parameter files or inter-notebook communication.

Set values in an Azure Databricks notebook by creating a parameters dictionary, using dbutils.jobs.taskValues.set to pass catalog, schema, and table names, and reference them dynamically across notebooks.

Discover how to manage Databricks notebooks using a single parameters dictionary with DBUtils widgets, convert string parameters with JSON.loads, and handle lists for looping tasks.

Create and configure SQL tasks in a pipeline by using SQL queries or SQL files, supplying parameters, and setting run conditions based on dependencies.