Azure Databricks & Spark for Data Engineers:Hands-on Project

learn to create a free Azure account, understand the 12-month free services and 200 USD credit, and compare student free accounts with 100 USD credit.

Sign in at portal.azure.com to explore the azure portal, navigate resources, dashboards, services, and use the search bar. Customize settings, manage subscriptions, and access co-pilot and cloud shell for navigation.

Explore how Databricks unifies data analytics with Apache Spark to build data lakehouses, featuring Photon, Unity Catalog, Delta Lake, and cloud integration on Azure, AWS, and Google Cloud.

Create an Azure Databricks workspace in portal by selecting a subscription, creating a resource group, choosing UK South and premium hybrid compute, then launch the workspace with Azure AD sign-in.

Master the Databricks user interface for data engineering by navigating the main menu, workspace, catalog, jobs and pipelines, compute options, and data ingestion workflows.

Explore the two-plane Databricks architecture, control plane and compute plane, and see how Web UI, cluster manager, Unity Catalog, and workspace storage enable data processing with classic and serverless compute.

Explore Databricks compute as the engine that runs notebooks, jobs, and pipelines. Learn to configure, create, and troubleshoot compute, addressing Azure quota and VM availability.

Discover Databricks compute options, including serverless and classic compute, and learn about clusters with driver and worker nodes, auto-scaling, execution-based billing, and workloads from etl to machine learning.

Explore classic compute configurations, choosing between single-node and multi-node clusters, and select access modes and runtimes to optimize Spark workloads, scalability, security, and cost.

Create and configure a Databricks cluster in the workspace, exploring single-node and multi-node setups, runtimes, autoscale, and termination controls for cost efficiency.

Diagnose databricks cluster creation issues by checking quotas and vm availability, then select an available node type (DS3v2 or F4) in your region and retry.

Explore Databricks notebooks as your primary development environment, learn magic commands to switch languages, use Databricks Utilities for files, parameters, and workflows, and master debugging techniques.

Explore Databricks notebooks, a Jupyter-style cluster-attached workspace with Python as default language and multi-language cells, plus markdown documentation, version history, and team sharing features.

Discover Databricks magic commands like %python, %sql, %md, %fs, %sh, %pip, and %run to build multi-language notebooks, manage data, and modularize code.

Explore Databricks utilities to run file system operations, secrets, widgets, and notebook workflows from Python, Scala, or R cells (not SQL), using dbutils.fs, display, and help to streamline notebooks.

Explore how to debug Databricks notebooks using the Python debugger, breakpoints, step in and step out, the debug console, and the variable explorer to fix Python cell code.

Discover Unity Catalog as the central governance layer in Databricks, enabling secure, role-based access to tables, views, and volumes across Azure data lake storage Gen2.

Explore the Unity Catalog object model, including metastore, catalogs, schemas, tables, views, functions, and volumes, and distinguish managed versus external tables with Delta Lake storage.

Access the unity catalog metastore configuration from the databricks account console and learn how to verify a workspace is attached using current_metastore, noting one metastore per Azure region.

Create a Unity Catalog metastore and attach your Databricks workspace to it, ensuring regional alignment and proper access settings via the Databricks account console.

Configure access to cloud storage with Unity Catalog by creating storage credentials and external locations tied to per-catalog containers, using managed identities or service principals and the Azure access connector.

Configure an Azure Databricks access connector, create an Azure Data Lake storage account, and assign the Storage Blob Data Contributor role to enable secure access from the data lake.

Configure access to cloud storage by creating a storage credential wrapped with the Unity Catalog access connector, then create an external location for the storage container.

Explore the Formula 1 project context, examine the datasets and their structure, and define the project requirements to build a solid foundation before designing the solution.

Explore the Formula 1 data landscape, including seasons, circuits, constructors, drivers, results, and sprint results. Learn how the Ergast data model uses csv and json formats to support race analysis.

Ingest six datasets from csv and json into Delta format, apply schema and audit columns, then transform to a gold layer enabling driver and constructor standings analysis with time travel.

Explore the data lakehouse foundation and medallion architecture, and walk through the overall solution architecture for the Formula One project, showing how the data pipeline layers come together.

Learn how data lakehouse blends the scalability of data lakes with the ACID governance of data warehouses. See how Delta Lake and Unity Catalog enable BI and machine learning workloads.

Explore medallion architecture and how bronze, silver, and gold layers enable incremental processing, governance, and high-quality data for analytics and reporting in a data lakehouse.

Explore a flexible medallion architecture for a Formula One project, from landing to bronze, silver, and gold layers, using delta format and Databricks jobs.

Set up the Azure data lake environment and Unity Catalog in Databricks to store and organize data for the Formula 1 project, then prepare to build your data pipelines.

Set up medallion architecture with landing, bronze, silver, and gold layers by creating formula1 container and landing folder in Azure Data Lake Storage, plus Unity Catalog location for 85 files.

Create the formula1 unity catalog with landing, bronze, silver, and gold schemas and a Files external volume, using a managed location tied to the formula1 container for Azure Data Lake.

Ingest all six F1 datasets from csv and json formats, apply schema and audit metadata, and store as delta bronze data using Spark DataFrame APIs for a production-ready ingestion workflow.

Load the circuits csv from the landing layer into a bronze delta table, enforcing or inferring schema, and add ingestion metadata like source file name and timestamp using PySpark.

Read circuits.csv with the DataFrame Reader API in PySpark to create circuits_df, then fix header handling with .format('csv') .load and .option('header', true) in Databricks, and preview the data with display.

Master how to define explicit schemas in PySpark with StructType and StructField, contrast with inferSchema, and explore read modes (permissive, dropMalformed, failFast) to handle bad records.

Add ingestion timestamp and source file metadata to the circuits dataset using Spark withColumn, enabling traceability and auditing before writing to the bronze layer.

Write circuits_final_df to a Delta table in the bronze schema using the DataFrameWriter API with full overwrite, via saveAsTable, and verify data using SQL and PySpark.

Ingest the races csv from the landing layer into bronze.races, define the schema, ensure correct data types, add ingestion metadata, and write to the bronze delta table.

refactor notebooks to remove hard-coded values and make them production-ready by using a common environment configuration notebook for catalog and bronze schema, applied to circuits and races.

Refactor repetitive notebook logic by extracting ingestion metadata into a helper function add_ingestion_metadata, then apply it across circuits and racers notebooks to add ingestion_timestamp and source_file, improving readability and maintainability.

Ingests the constructors.json from landing, applies a DDL-style schema with four string columns (constructor_id, name, nationality, url), adds ingestion metadata, and writes to the bronze.constructors Delta table.

Ingest the drivers dataset into the bronze layer with the Spark DataFrame Reader API, preserving the nested name struct and writing to the bronze delta table with metadata.

Ingest the results dataset from a folder into the bronze layer with Spark, reading all JSON files in the folder path and writing to Delta table with a metadata column.

Ingest multi-line sprint JSON files into the bronze layer across five years (2021–2025). Use the DataFrameReader API and set the multiLine option to true, then write to a Delta table.