Microsoft Fabric Masterclass : Real-Time Data Engineering

Name: Microsoft Fabric Masterclass : Real-Time Data Engineering
Rating: 4.9 (12 reviews)

Learn Microsoft Fabric for real-time data engineering: pipelines, dataflows, streaming, and KQL queries.

Created byMallaiah Somula

Last updated 5/2026

English

English [Auto],

What you'll learn

Understand Microsoft Fabric architecture, components, and core services.
Build end-to-end data pipelines for cloud and on-premise data ingestion.
Perform incremental and batch data loads using SQL DB, ADLS, SFTP, REST APIs, and Excel.
Design reusable ingestion frameworks with concurrency, dependencies, and scheduling.
Use Dataflow Gen2 for profiling, cleaning, and transforming large datasets.
Implement real-time streaming pipelines with Eventstreams, Event Hubs, and KQL.
Apply best practices in Fabric for data engineering, storage, and governance.

Course content

13 sections • 35 lectures • 38h 12m total length

Hands-On: Introduction to Microsoft Fabric: Architecture & Components1:01:58
Explore Microsoft Fabric's architecture and components, including its unified one lake, lakehouse, data factory, Synapse, Kusto, and Power BI, as an all-in-one SaaS data platform.

Hands-On : Copying Multiple Folders from ADLS Gen2 to Lakehouse1:00:10
Learn to copy data from adls gen2 to a fabric lakehouse using fabric pipeline, including configuring connections, mapping source and destination, and using wildcard paths for multiple files.
Hands-On : Using Lookup & ForEach for Parallel Ingestion with Retries1:08:25
Demonstrate real-time data ingestion in Microsoft Fabric by using a configuration table with lookup and for each to copy from blob storage to Lakehouse, with retries.

Hands-On : Copy Multiple SQL Tables to Warehouse & Archive in Lakehouse1:01:49
Copy multiple SQL tables from the SQL database to the warehouse and archive the data in Lakehouse, using zip-compressed archival files to cut storage costs.
Hands-On : Incremental Data Load from SQL Database to Lakehouse1:03:43
Learn to implement incremental data load from Azure SQL database to lakehouse with Microsoft Fabric, using a watermark and last processed value in a lookup, copy, and stored procedure pipeline.
Hands-On : Copy SQL Data using Tables, Queries & Stored Procedures1:01:00
Copy data from an Azure SQL database to lakehouse using tables, queries, and parameterized stored procedures, exploring dynamic query options and incremental loads with delta parquet files.

Hands-On : Incremental Loading of Multiple SQL Tables into Lakehouse1:01:05
This microsoft fabric masterclass demonstrates incrementally loading data from multiple sql tables into a lakehouse. It uses a watermark and config table, max-date logic, and per-table copy activity.
Hands-On : Extracting and Storing Pipeline Logs in Fabric Warehouse1:08:38
Extract and store pipeline logs in Fabric by copying data from Azure SQL database to lakehouse and log run id, status, duration, and errors in a Fabric SQL audit table.

Hands-On : Ingest On-Prem SQL Data using Data Gateway59:44
Ingest on-prem SQL data into a Fabric warehouse using a data gateway and a data pipeline. Build and configure the on-prem SQL server, gateway, and configuration table for multi-table syncing.
Hands-On : Ingest Files from On-Prem File System to Lakehouse59:39
Learn to copy data from on premise file system to lakehouse using a data gateway, covering gateway setup, workspace configuration, and a script-driven for-each data pipeline.

Hands-On : Ingest Incremental Files from SFTP into Lakehouse1:48:16
Learn to ingest incremental files from SFTP to lakehouse with Microsoft Fabric, using a config table and lower and upper time bounds to track progress.
Hands-On : Ingest Incremental Files from ADLS Gen2 into Lakehouse58:26
Learn to ingest incremental files from ADLS Gen2 into a lakehouse using Microsoft Fabric. Build a dynamic, incremental load pipeline with UTC/IST conversions, config-driven control, and merged output.

Hands-On : Ingest Data from Snowflake into Fabric Warehouse1:16:33
Copy data from Snowflake into Fabric warehouse while evaluating shared disk vs shared nothing architectures, and explore Snowflake's scalable storage and compute separation for real-time data ingestion.
Hands-On : Ingest REST API Data with Dynamic Bearer Tokens55:12
Ingest data from a REST API to a lakehouse in this hands-on Microsoft Fabric Masterclass, using dynamically generated bearer tokens per run passed to the copy activity.

Hands-On : Ingest Multi-Sheet Excel Data into Unified Tables1:11:36
Learn to ingest Excel data from multiple stores into a unified fabric table with dynamic schema using get metadata and for each, add source name, file name, and load date.
Hands-On : Convert Excel Sheets into Tables using Dynamic Sheet Index58:41
Master how to ingest multiple Excel sources with different schemas into a unified warehouse table using dynamic sheet index mappings in Microsoft Fabric masterclass.

Hands-On : Build a Reusable Ingestion Framework Pipeline1:03:16
Learn to build a reusable ingestion framework pipeline in Microsoft Fabric that ingests data from SQL DB, ADLs, and blob into Lakehouse using parameterized pipeline, switch logic, and child pipelines.
Hands-On : Scheduling Logic for Ingestion Pipelines1:06:29
Demonstrates building and scheduling a reusable ingestion framework in fabric, using a configuration-driven approach to copy data from SQL databases and ADLs to a lakehouse via scheduled pipelines.
Hands-On : Controlling Pipeline Concurrency with Control Tables1:09:31
Learn to control pipeline concurrency with a control table in Microsoft Fabric, preventing parallel runs by checking in-progress status, and skipping or invoking pipelines via script and for-each.
Hands-On : Multi-Trigger Dependency for Pipelines55:25
Master multi-trigger dependency for ingestion pipelines in Microsoft Fabric, coordinating on-premise loads, blob storage success files, and lakehouse copy via until checks.
Hands-On : Using Invoke Pipeline Activity for Multi-Trigger Execution1:14:17
Explore how to coordinate a multi-trigger data pipeline in microsoft fabric by verifying the first three success files and invoking the ingestion pipeline with the invoke pipeline activity.
Hands-On : Auto-Delete Old or Small Files from Storage1:00:21
Learn to auto-delete small files and data older than a set window in data lake fabric pipelines, using get metadata, for each, delete, logging, and soft delete recovery.

Requirements

No prior experience with Microsoft Fabric is required — this course starts from the basics.
Familiarity with SQL queries will be helpful for hands-on exercises.
Basic understanding of data concepts (databases, files, pipelines) is recommended.
An active Microsoft Fabric trial account (or organizational account) for practice.
A computer with internet connection to follow along with labs and exercises.

Description

Unlock the full potential of Microsoft Fabric with this hands-on masterclass designed for data engineers, analysts, and architects. This course takes you step by step through real-world scenarios, helping you build pipelines, ingest and transform data, work with Dataflows Gen2, stream real-time data, and query with KQL.

You’ll start with the fundamentals of Microsoft Fabric architecture and progress into advanced data engineering techniques such as incremental loading, mirroring databases, building reusable ingestion frameworks, and setting up multi-trigger dependencies.

Through 34 practical, project-based lessons, you’ll learn how to:

Connect to diverse data sources (SQL, Snowflake, REST APIs, SFTP, Blob Storage, Excel).
Design and optimize pipelines for efficiency, parallelism, and incremental loads.
Work with Lakehouse and Warehouse for structured and unstructured data.
Perform data quality checks, profiling, and cleansing using Dataflows Gen2.
Stream real-time data from Event Hubs to Fabric Eventhouse and KQL databases.
Implement best practices for scheduling, orchestration, and monitoring pipelines.

By the end of this course, you’ll have built a portfolio of end-to-end Microsoft Fabric projects that prepare you for solving real-world data challenges at scale.

Whether you’re a beginner exploring Microsoft Fabric or a professional looking to upskill in modern data engineering, this course will give you the confidence and expertise to design and deliver data solutions using Microsoft Fabric.

Who this course is for:

Data Engineers who want to design and build modern data pipelines in Microsoft Fabric.
Data Analysts & BI Developers looking to leverage Fabric’s Lakehouse, Warehouse, and Power BI integration.
Cloud Professionals transitioning from Azure Synapse, Databricks, or traditional ETL tools to Fabric.
Students & Beginners interested in learning real-time data engineering without prior Fabric experience.
Enterprise Teams who want to adopt Microsoft Fabric for end-to-end data management and analytics.

Microsoft Fabric Masterclass : Real-Time Data Engineering

What you'll learn

Explore related topics

Course content

Introduction to Microsoft Fabric1 lecture • 1hr 2min

Data Ingestion from Azure Storage2 lectures • 2hr 9min

SQL Database Ingestion3 lectures • 3hr 7min

Advanced SQL Ingestion & Monitoring2 lectures • 2hr 10min

Hybrid & On-Premises Data Integration2 lectures • 1hr 59min

Incremental File-Based Ingestion2 lectures • 2hr 47min

External Cloud & API Integration2 lectures • 2hr 12min

Excel Data Ingestion2 lectures • 2hr 10min

Enterprise Ingestion Frameworks6 lectures • 6hr 29min

REST API Advanced Integration1 lecture • 1hr 10min

Requirements

Description

Who this course is for: