Databricks Certified Data Engineer Associate - Ultimate Prep

Learn how to create a free azure account, explore 12 months of free services and 200 USD of free credit, and compare with the student free option and sign-up steps.

Navigate the Azure portal to sign in, access resources, dashboards, and recently used items, using the search bar and Copilot for quick answers, and explore Cloud Shell and portal settings.

Explore how Databricks enables a modern data lakehouse that blends data lake flexibility with data warehouse governance, supporting BI and ML workloads on a unified platform.

Discover how medallion architecture structures data as bronze, silver, and gold layers in a Databricks data lakehouse, enabling quality, governance, lineage, incremental processing, and role-based access control.

Explore how Databricks integrates Spark for building data lake houses, featuring photon, Delta Lake, Unity Catalog, and Delta Live Tables across Azure, AWS, and Google Cloud.

Create an Azure Databricks workspace in the Azure portal by selecting a subscription, resource group, and region, enable premium features including Delta Live Tables, Unity Catalog, and role-based access control.

Explore the Databricks user interface, focusing on the left navigation by product areas, key menus (workspace, catalog, jobs, compute, data ingestion), and the search bar to navigate the workspace confidently.

Explore the two main Databricks planes—control plane and compute plane—and how classic and serverless compute, Unity Catalog data governance, and workspace storage locate resources and data.

Explore Databricks compute options, from serverless to classic, as clusters powered by driver and worker nodes run ETL, data science, and machine learning workloads.

Configure databricks clusters by choosing single or multi-node setups, access modes, runtimes, and vm types to balance performance and cost. Use cluster policies to enforce safe shared compute.

Create and configure a single-node Databricks compute in the workspace, selecting runtime, node type, auto-termination, and optional advanced options. Learn to monitor clusters via event logs and manage cluster details.

Debug Databricks cluster creation by resolving quota and VM availability in your region, selecting a valid node type for a single-node cluster, and checking Azure quotas to restart successfully.

Explore Databricks notebooks, a Jupyter-style environment that runs Python, SQL, and Scala on a cluster, with markdown documentation, magic commands, and seamless workspace organization.

Learn how Databricks magic commands switch cell languages, create markdown, manage the file system, run shell commands, install libraries with pip, and include notebooks with run.

Explore Databricks utilities to combine file operations with ETL tasks in notebooks, using dbutils.fs and the percentage fs magic, plus secrets, widget, and workflow utilities.

Databricks git folders offer a visual git client in the workspace that bridges notebooks and scripts to GitHub, Azure DevOps, and Bitbucket, enabling branches, reviews, and pull requests.

Learn to link GitHub with the Databricks workspace, create and clone a repository, and use git folders with feature branches, commits, pushes, and pull requests for deployment readiness.

Enable the Python debugger, set breakpoints, step through code, inspect variables, and use the debug console to fix logic errors in Databricks notebooks.

Explore Unity Catalog, Databricks' data governance solution, and configure Unity Catalog objects for data lakehouse access. Understand legacy Hive Metastore and the unified object model for secure cloud storage access.

Navigate the Unity Catalog object model from Metastore to catalogs, schemas, volumes, and tables, contrast managed versus external data, and understand three-level namespaces and Hive Metastore compatibility.

Create a unity catalog metastore in your Azure region and attach your workspace. Verify the attachment with a SQL command and assign metastore admin privileges.

Configure Databricks clusters for Unity Catalog by selecting runtime 11.3 or higher, choosing an appropriate access mode, and disabling credential pass through, ensuring the workspace has Unity Catalog enabled.

Configure access to cloud storage using Unity Catalog by creating storage credentials and external locations, with a managed identity or service principal, and applying granular access per catalog or schema.

Configure access to Azure data lake storage through Unity Catalog by creating an access connector and storage credential, then define an external location and verify access from a Databricks notebook.

Learn to design and implement ETL pipelines with Apache Spark, validate and transform data from bronze to silver to gold in a data lakehouse, using Spark SQL and PySpark.

Develop a simple ETL project with Apache Spark to build a Gizmo Box data lake house, covering bronze, silver, and gold layers, Unity Catalog, and external vs internal data workflows.

Set up the data lake project environment by creating Gizmo Box container, organizing landing data into operational and external folders, uploading files, and granting Databricks access via an external location.

Set up Unity Catalog project environment with Gizmo Box, landing, bronze, silver, and gold schemas, and an operational data volume; learn to create catalogs, schemas, volumes, and manage permissions.

Extract customers data from the landing layer by querying JSON files with Spark SQL, using single files, wildcards, and folders; audit data sources with file path techniques and underscore metadata.

Create a Unity Catalog view in the bronze schema to reference landing-layer data, using a three-level namespace and create or replace view, enabling lineage, security, and easy data access.

Create temporary views and global temporary views in Spark within Databricks, and understand their lifecycles for use in intermediate ETL results across notebooks.

Extract orders data from a complex json with nested arrays and data quality issues using text format for pre-processing before json parsing, then create a bronze view for raw data.

Process unstructured membership image files using the binary file format in Databricks. Query with select, view the data schema, and access metadata across subfolders of PNG identity cards.

Learn to read tab-delimited csv addresses data with the read_files function, including header handling and delimiter specification. Compare select limitations with alternatives like external tables and bronze layer.

Extract payments data from a csv stored in Azure Data Lake using an external table in the bronze schema with Unity Catalog, accessed via ABFS without moving files.

Learn to access refunds data from an Azure SQL database in Databricks by creating an external table via JDBC in Hive metastore, then query the external table.

Learn to run SQL commands from Python using Spark SQL, read JSON and other formats with the DataFrame Reader API, and create temporary views with spark.table.

Explore data profiling in Databricks using the notebook UI and dbutils, compare programmatic summaries, and identify nulls and duplicates to address data quality issues.

Address data quality issues in the customers data by removing null IDs, deduplicating records, casting timestamp and date fields, and writing the cleaned data to the silver layer.

Transform payments data by extracting and splitting the payment timestamp into date and time. Decode statuses with a case expression and create a silver table in unity catalog.

Transform refunds data by splitting refund reason and source using split and regexp_extract, extract date and time from the refund timestamp, and write results to hive metastore silver layer.

Transform memberships data by extracting the customer id from the file path using regexp_extract. Create a silver memberships table and join it to the customers table for integrated insights.

Transform addresses data by pivoting two rows into a single customer row using the pivot class with address type and the for clause to create shipping and billing fields.

Extract information from json strings in the orders data using extraction path and array indexing. Use dot notation for fields and cast as needed, noting performance limits of string reads.

Transform orders data by converting json strings to json objects, fixing data quality issues with pre-processing, and building a sql table in the silver schema for downstream processing.

Access json order data using dot notation and flatten arrays with explode. Deduplicate with array_distinct, extract nested fields, and write a silver table for downstream analytics.

Learn how to inner join customers and addresses on customer_id, pivot shipping and billing addresses into a single row, and create a gold layer customer address table for downstream use.

Master spark aggregate functions to summarize orders by customer and month, calculating total orders, total items bought, and total amount spent using price times quantity.

Explore Spark user defined functions (UDFs) to encode business logic with SQL (preferred) or Python, stored in Unity Catalog or Hive Metastore for reuse across queries.

Explore higher order functions that operate on arrays and maps, using lambdas to transform, filter, exist, and aggregate; see examples with named structs and total order calculations.

Explore higher order functions for maps, including transform values, transform keys, and map filter, with examples converting keys to uppercase, applying 10% tax, and filtering items above 500.

Explore PySpark, the Python API for Apache Spark, and learn how data frames enable flexible, programmable ETL from data sources through the DataFrame API, from read to transform to write.

Convert Gizmo Box extract customers data workflow from Spark SQL to PySpark, read JSON with DataFrame Reader API, and write to a Delta table using DataFrame Writer v2 API.

extract orders data from a json file by reading as text to handle corrupt records, then write json strings to pie.orders in the gizmo box catalog using py spark v2.

Extract memberships data from binary image files using PySpark, read all PNGs, and write to gizmo box bronze p y underscore memberships table in target schema via DataFrame Writer v2.

Extract addresses data from the addresses folder and load it into a Spark table using the data frame reader api with csv, header, and tab delimiter.

Extract payments data from csv using the data frame reader API in PySpark, define the schema (ddl format or python format), and write to a table with writer v2.

Extract refunds data from an Azure SQL table using Spark data frame reader with JDBC, then write it to a bronze Delta Lake table via the data frame write API.

Transform the bronze customer data into a silver table using PySpark by cleaning nulls and duplicates, keeping latest by created timestamp, and writing to the silver table.

Transform payments data by extracting date and time from the payment timestamp, translating numeric statuses to text, and writing the results to the silver layer.

Transform refunds data from bronze to silver by extracting refund reason and source with split or regex, and parsing the date and time from the refund timestamp using PySpark.

Transform bronze-layer membership data to the silver layer by extracting the customer id from the file path and writing the results to the silver memberships table.

Denormalize addresses from bronze to silver by pivoting on address type after grouping by customer id, aggregating with max for address line, city, state, postcode to a single customer record.

Transform orders data by converting JSON strings to JSON objects, fixing data quality issues from bronze to silver using regexp_replace and from_json in Spark.

Access the orders JSON, deduplicate the items array with array_distinct, explode it into individual item records, and write the transformed data to the silver layer using PySpark.

Join the silver customers and silver addresses to create the gold table customer_address in the gold schema using PySpark data frame joins on customer_id and write to a delta table.

Read from silver orders table, add order month with column, group by order month and customer ID to compute total orders, total items, and total amount; write to gold table.