Microsoft Fabric Data Engineering Mastery: A Complete Guide

Medallion Architecture is a layered data architecture designed for handling large volumes of data.  It organizes data into multiple layers to improve data quality and accessibility.

Learn about the raw, bronze, silver, and gold data layers.

After completing this lecture you know about some common data models used to store and report data.  We'll look and each model's characteristics and the commons uses for each.

By the end of this lesson you'll have created a free trial to use Microsoft Fabric.

Create a Workspace within Microsoft Fabric.  An MS workspace is a collaborative environment in Microsoft Fabric that provides a unified space for managing, developing, and sharing datasets, reports, and other data-related assets across teams.

We create a lakehouse to store data.  A raw data layer Lakehouse in MS Fabric is a foundational storage layer where unprocessed, raw data is ingested and stored in its original format, serving as the starting point for data processing and transformation workflows within the lakehouse architecture.

You can download the Car Prices data set from the resource.  I also included a link to the data set on Kaggle.

In this lecture you'll learn how to prepare data from the bronze layer using Dataflow Gen2 low code methods.

In this lesson we will go over the data transformations necessary to move the car sales data from the bronze to silver layer.

Learn to use MS Fabric Dataflow, the low code data ingestion method, to transform data from bronze to the silver layer.

Learn to use MS Fabric Dataflow, the low code data ingestion method, to transform data from bronze to the silver layer.

Learn how to troubleshoot data refresh errors.

In this lesson we'll look at some of the specific steps to take to move the data to the gold layer.  We'll work with star schema dimensions and a fact table to make our data easily reportable.

In this lecture we'll fix some data issues.  Rather than fix them in the gold layer, we'll go back to the Silver, where this fix should have been done, and make changes and refresh the lakehouses.

In lessons part 1-3 learn how to take the flattened data from the silver layer to create dimensions.  The dimension are then match back to the silver layer data, to create the final fact table.  The end result is a star schema with the fact table referencing the dimensions via their ID's.

In lessons part 1-3 learn how to take the flattened data from the silver layer to create dimensions.  The dimension are then match back to the silver layer data, to create the final fact table.  The end result is a star schema with the fact table referencing the dimensions via their ID's.

In lessons part 1-3 learn how to take the flattened data from the silver layer to create dimensions.  The dimension are then match back to the silver layer data, to create the final fact table.  The end result is a star schema with the fact table referencing the dimensions via their ID's.

Setup the gold layer lakehouse's semantic layer.  We'll setup the relationships between the dimensions and fact table.

Learn to use PowerBI reporting to visualize the gold layer data.

Now that we see some issue with the data, let go back to the appropriate data layer and fix.  This will help tighten up the dimensions and make reporting more accurate.

Learn how to create a simple pipeline to automate the Bronze, Silver, and Gold layer refresh.

Learn to ingest and transform data using SQL.  We'll use the following MS Fabric concepts:  lakehouse, data pipelines, synapse data warehouse, T-SQL, and stored procedures.

Setup a new workspace for the SQL case study.  We'll setup a lakehouse and data warehouse.

Set up a data pipeline to copy data from the Raw data layer to Bronze layer.  You'll learn how to use the Copy Activity to read a CSV file and convert it to parquet.

The goal for this series of lectures is to take our raw car sales and get the ready for use in the silver layer.

In Lessons 1-3, we will dive into essential SQL data engineering practices. You’ll learn how to transform data from the Bronze to Silver layers, focusing on parsing key fields like the sales date. We’ll also cover the creation of stored procedures to efficiently process data and build automated pipelines to streamline these processes.

In Lessons 1-3, we will dive into essential SQL data engineering practices. You’ll learn how to transform data from the Bronze to Silver layers, focusing on parsing key fields like the sales date. We’ll also cover the creation of stored procedures to efficiently process data and build automated pipelines to streamline these processes.

In Lessons 1-3, we will dive into essential SQL data engineering practices. You’ll learn how to transform data from the Bronze to Silver layers, focusing on parsing key fields like the sales date. We’ll also cover the creation of stored procedures to efficiently process data and build automated pipelines to streamline these processes.

In this lesson, you'll learn how to use Azure Data Studio alongside MS Fabric to streamline your data engineering tasks.

The goal of this series of lectures is to transition car sales data from the Silver layer to construct a star schema in the Gold layer. The star schema features a simple, denormalized design optimized for query performance, ideal for business intelligence and reporting. The process includes identifying and building fact and dimension tables, such as ModelYear, Make, and SalesDate for dimensions, and sales-related metrics like SellingPrice and SalesDate for facts, to facilitate comprehensive data analysis and reporting.

In part 1, 2, and 3 you will learn how to use SQL and SQL Stored Procedures in Synapse Data Warehouse to build dimensions for a star schema. Activities will include identifying and defining dimensions such as ModelYear, Make, and SalesDate, and mapping these dimensions to their attributes, like SellerName and Color. You'll also learn to organize these dimensions effectively to ensure they integrate seamlessly with fact tables and optimize query performance.

In part 1, 2, and 3 you will learn how to use SQL and SQL Stored Procedures in Synapse Data Warehouse to build dimensions for a star schema. Activities will include identifying and defining dimensions such as ModelYear, Make, and SalesDate, and mapping these dimensions to their attributes, like SellerName and Color. You'll also learn to organize these dimensions effectively to ensure they integrate seamlessly with fact tables and optimize query performance.

In part 1, 2, and 3 you will learn how to use SQL and SQL Stored Procedures in Synapse Data Warehouse to build dimensions for a star schema. Activities will include identifying and defining dimensions such as ModelYear, Make, and SalesDate, and mapping these dimensions to their attributes, like SellerName and Color. You'll also learn to organize these dimensions effectively to ensure they integrate seamlessly with fact tables and optimize query performance.

Automate load from Raw to Gold using MC Fabric data pipelines.

Hook a PowerBI report into the semantic model to visualize the gold layer data from within the data warehouse.

In these lessons, you'll dive into how to leverage PySpark and Python for efficient data ingestion and transformation. We'll cover the architecture of the Lakehouse, including the Raw, Bronze, Silver, and Gold layers, and explore key technologies such as PySpark DataFrames, Lakehouse, Synapse Notebooks, and Data Pipelines. By the end, you'll understand how to effectively utilize these tools to manage and transform data across different stages of the data pipeline.

In this lesson  you'll set up the lakehouses needed to move data from the raw layer to gold using the PySpark library.

In this lesson, we'll learn how to move data from the RAW to the Bronze layer using a Python notebook.  We'll read a CSV file from the RAW Lakehouse, add tracking columns to capture file path and date, and then save the data as a Parquet file in the Bronze layer. This process ensures our data is properly formatted and ready for the next stages of processing.

In parts 1-3, you'll learn how to transform data from the Bronze to the Silver layer using PySpark in Synapse Notebooks. We’ll read data from the Bronze layer, perform essential data cleaning tasks such as renaming columns, fixing data types, and standardizing formats. Key transformations include adjusting the format of date fields, handling condition values, and dropping unnecessary columns. Finally, we'll save the refined data as a Delta table in the Silver layer, optimizing it for advanced analysis.

In parts 1-3, you'll learn how to transform data from the Bronze to the Silver layer using PySpark in Synapse Notebooks. We’ll read data from the Bronze layer, perform essential data cleaning tasks such as renaming columns, fixing data types, and standardizing formats. Key transformations include adjusting the format of date fields, handling condition values, and dropping unnecessary columns. Finally, we'll save the refined data as a Delta table in the Silver layer, optimizing it for advanced analysis.

In parts 1-3, you'll learn how to transform data from the Bronze to the Silver layer using PySpark in Synapse Notebooks. We’ll read data from the Bronze layer, perform essential data cleaning tasks such as renaming columns, fixing data types, and standardizing formats. Key transformations include adjusting the format of date fields, handling condition values, and dropping unnecessary columns. Finally, we'll save the refined data as a Delta table in the Silver layer, optimizing it for advanced analysis.

In this lesson, we will focus on creating dimension tables for our car sales data in the Gold Layer. We'll start by identifying key attributes such as ModelYear, Make, and Transmission, and then build corresponding dimension tables to represent these attributes. Using PySpark, we’ll ensure the data is cleaned, transformed, and properly structured. This process will enhance query performance and make the data more accessible for reporting and analysis.

In this lesson, we will focus on creating dimension tables for our car sales data in the Gold Layer. We'll start by identifying key attributes such as ModelYear, Make, and Transmission, and then build corresponding dimension tables to represent these attributes. Using PySpark, we’ll ensure the data is cleaned, transformed, and properly structured. This process will enhance query performance and make the data more accessible for reporting and analysis.

In this lesson, we will focus on creating dimension tables for our car sales data in the Gold Layer. We'll start by identifying key attributes such as ModelYear, Make, and Transmission, and then build corresponding dimension tables to represent these attributes. Using PySpark, we’ll ensure the data is cleaned, transformed, and properly structured. This process will enhance query performance and make the data more accessible for reporting and analysis.

In this lesson, we will focus on constructing the fact table for our car sales data in the Gold Layer. We’ll gather and aggregate quantitative metrics such as SellingPrice, Miles, and ManheimMarketReportValue, and organize these into a fact table that captures key sales performance indicators. Using PySpark, we’ll ensure the fact table is properly linked with dimension tables and optimized for efficient querying. This will provide a solid foundation for detailed analysis and reporting.

Construct a pipeline to automate Python notebook execution to load the Bronze, Silver, and Gold Layers.

Add tables to the default semantic model, then add relationships between dimensions and the Car Sales fact table. 

Use PowerBI reports to visualize data represented via the gold layer semantic model.