DP-750: Azure Databricks Data Engineer Associate Exam Prep

Learn how Azure Databricks fits into the Azure ecosystem, from real-time ingestion and ADLS storage with Delta Lake to gold layer, ML models, and governance with Purview and Unity Catalog.

Deploy an Azure Databricks workspace in the portal, configure resource groups, region, and a trial pricing tier, and explore the workspace UI focused on data engineering, compute, and Unity Catalog.

Understand Apache Spark's distributed execution across driver and worker nodes. Explore cloud data sources, data frames, and catalyst optimizer for Python and sql workflows.

Deploy a managed all-purpose compute instance in the Databricks workspace to run notebooks with the Spark runtime libraries. Configure runtime version and cost controls before creating dp750-compute.

Assess compute targets for Azure Databricks by distinguishing interactive notebooks from automated pipelines and selecting serverless, managed, SQL warehouse, or job compute, mapped to memory-optimized, compute-optimized, storage-optimized vm families.

Clone the public dp750 GitHub repo into a Databricks git folder, link it to the Databricks workspace, and set up serverless or all-purpose compute (Spark 4.0.0) for notebooks and labs.

Learn how the Unity Catalog in Azure Databricks organizes data with catalogs, schemas, tables, volumes, and a metastore, and how it maps to Azure Data Lake storage.

Join a hands-on lab to manipulate data with Apache Spark in Unity Catalog, build schemas on Spark data frames, run Spark and SQL queries, and visualize results in Azure Databricks.

Work through advanced spark operations in a hands-on lab, building and cleaning dataframes from CSV files in a unity catalog, applying schema, tax calc, name splitting, and year-by-item aggregations.

Perform delta table operations in Databricks by loading a CSV into Delta format, enabling versioning, Unity Catalog governance, and delta history for AI/ML workloads.

Explore the medallion architecture from raw data to business aggregates using bronze, silver, and gold delta tables, implemented with spark declarative pipelines in production.

Implement the medallion architecture with Delta Lake in Spark by building bronze, silver, and gold layers in Unity Catalog, enforcing schemas, cleaning data, and deriving yearly sales aggregates.

Understand views in Databricks: compare materialized and non-materialized views, learn how derived tables from complex queries improve performance, freshness, maintenance, and secure sharing via Unity Catalog.

Learn how the AI/BI genie in Databricks, a foundation large language model, queries structured data in natural language and generates SQL from table relationships with metadata and access controls.

Use materialized and non-materialized views with the ai and bi genie to run natural language queries, attach views to a genie space, and visualize results on a serverless starter warehouse.

Create an Azure data lake storage account with hierarchical namespace, attach the system assigned managed identity, and enable Unity Catalog integration with an external location in Databricks.

Create a Unity Catalog on your ADLS in the Databricks workspace, naming it DP750 Demo Catalog, and link it to the external location to store delta tables and volumes.

Discover how Azure Key Vault enables credential-less access for Azure Databricks notebooks by storing secrets, using a service principal and RBAC, and registering a Delta table in the Unity catalog.

Learn to integrate azure key vault with azure databricks by provisioning a key vault, configuring rbac permissions, creating username and password secrets, and accessing them through a databricks secret scope.

Explore row filtering and column masking in Azure Databricks to enforce table-level and view-level security, with practical use cases for sales, HR, and regional data access.

Implement row-level and column-level security on a delta table using Unity Catalog, creating and applying user-defined masking and filtering functions to protect emails and restrict rows.

Apply row-level filtering and column masking on a non-materialized view derived from the sales raw delta table, demonstrating view-level security versus table-level access in Unity Catalog.

Learn how file formats (CSV, JSON, Parquet) underpin table formats (Delta Lake, Apache Iceberg) in Azure Databricks, enabling ACID transactions, metadata tracking, and efficient Spark SQL on Parquet data.

Learn to create and manage managed and external delta tables in Azure Databricks, compare their lifecycle of data and metadata, and explore Unity Catalog external locations and ADLS integration.

Explore data partitioning in Azure Databricks by partitioning a delta table on sign up year, loading a csv demo, and measuring query performance with where filters on sign up year.

Discover how the vacuum command in Azure Databricks removes unused Delta files to save storage, disable retention checks, and its impact on time travel and version revert.

Tackle performance tuning in Azure Databricks by simulating a small file problem, using optimize to consolidate into one file, and vacuum to remove unused files while exploring time travel implications.

Learn how z-ordering and liquid clustering optimize delta tables in azure databricks by co-locating data within partitions and adapting to query patterns.

Improve delta table performance in Azure Databricks by applying z-ordering and liquid clustering, using partitioning by city, optimize commands, and analyzing file counts to boost query speeds.

Understand the star schema model, with a central fact table and linked dimension tables via primary and foreign keys, describing products, employees, and time for sales analytics.

Demonstrate change data feed and audit trail in Azure Databricks using Delta tables in Unity Catalog, with update pre-image and update post-image and time travel.

Explore the Azure Databricks data ingestion landscape by examining sources and behaviors, and design pipelines using Unity Catalog with tools like Spark Structured Streaming, change data capture, and Autoloader.

Understand how Lakeflow Connect enables data ingestion from multiple SaaS tools in Azure Databricks with fully managed connectors, drag-and-drop configuration, and bronze-to-gold medallion architecture for BI and AI/ML workloads.

Use Databricks managed and standard connectors to ingest data from azure data lake storage, sql server, and salesforce, setting ingestion mode to one-time or scheduled in Unity Catalog managed table.

Leverage Databricks notebooks to prototype data ingestion with PySpark and the DataFrame API, perform transformations, and store Delta tables across bronze to gold layers for BI, AI, and ML workloads.

discover how to ingest data with Databricks notebooks, create a Unity Catalog EV_lab, land raw csvs in the bronze raw files volume, enforce schemas, and load into a Delta table.

Use create table as select to ingest data in Azure Databricks from existing delta or parquet sources in Unity Catalog, applying transformations and enabling refresh with create or replace table.

Showcases CTAS to ingest a Unity Catalog CSV into a delta bronze table with casting, then builds a silver table of EV charging sessions featuring duration and efficiency metrics.

Discover how the copy into sql command enables reliable, incremental data ingestion from cloud storage into a delta table in azure databricks, tracking ingested files with a transaction log.

Learn to ingest CSV data into Azure Databricks using copy into, building bronze and silver tables in Unity Catalog, with idempotent batch processing and schema inference.

Learn to use Spark Structured Streaming for real-time data ingestion in Azure Databricks, connecting to Azure Event Hub, Kafka, or AWS Kinesis, and processing with the DataFrame API.

Ingest real-time data with Spark Structured Streaming on Azure Databricks, configuring a Unity Catalog input path and checkpoint to stream CSV files into a bronze Delta table.

Explore autoloader for incremental data ingestion from cloud storage into Delta tables in Unity Catalog on Azure Databricks, with on-the-fly schema evolution, event-driven loading, and checkpointed processing to avoid duplicates.

Explore autoloader vs copy into for data ingestion in Azure Databricks, using a decision framework for streaming, batch, and schema evolution with Unity Catalog delta table.

Leverage auto loader for real-time data ingestion in Azure Databricks, using unity catalog volumes, schema drift management with merge schema, and writing to a bronze delta table with checkpointing.

Select the right Azure Databricks data ingestion tool using a decision framework based on data source type, including Lakeflow Connect Managed Connectors and autoloader with Delta tables in Unity Catalog.

Explore data profiling and statistical insights in Azure Databricks to understand data structure, quality, and delta tables, and enable Unity Catalog profiling in three modes: snapshot, time series, and inference.

Learn to profile data and generate statistical insights for delta tables in Azure Databricks using SQL and PySpark, including full-table statistics and column-level analytics in Unity Catalog.

Explore simple and complex data types, including date and timestamp with or without time zone, arrays, maps, and structs, and apply null handling and deduplication in Unity Catalog delta tables.

In this hands-on lab, learn to work with simple and complex data types, handle nulls and duplicates, and move data from bronze to silver using delta tables and medallion architecture.

Learn PySpark and SQL filtering and aggregation on the bronze orders delta table, filtering by order value and delivered, then computing city revenue and cuisine counts, with null delivery times.

Explore inner, left, right, full, semi, anti, and cross joins and set operators in Azure Databricks, and how they join central fact tables with dimension tables in star schemas.

Explore normalization and denormalization in a star schema, showing how a lean fact table uses keys to join with dimensions, and when to pivot or unpivot for dashboards.

Pivot and unpivot data in Azure Databricks with SQL to transform the orders and payments tables, creating cuisine columns and a metric of amounts in a hands-on lab.

Learn to design automated data pipelines in Azure Databricks using notebooks and Spark declarative pipelines, moving data from bronze to silver to gold in Unity Catalog with a hybrid approach.

Learn to parameterize Azure Databricks pipelines using widgets to pass city and minimum revenue, generating parameterized gold-layer tables from the transactions enriched data.

Explore how version control software like Git enables collaboration on code and notebooks, and how Azure Databricks integrates Git to manage notebooks and infrastructure as code dev, test, and prod.

Learn to use git in azure databricks workspace, configure credentials, create a demo dev branch from main, commit changes, push to GitHub, and raise a pull request.

Create and deploy a declarative automation bundle in a Databricks workspace using YAML-based infrastructure as code, including a notebook and a Spark ETL pipeline.