Mastering Azure Data Factory : Real-World Projects With ADF

Name: Mastering Azure Data Factory : Real-World Projects With ADF
Rating: 4.6 (35 reviews)

Build Real-Time Data Pipelines with Azure Data Factory, SQL, REST. From Beginner to Expert: Real Projects That Build

Created byMallaiah Somula

Last updated 7/2026

English

English [Auto],

What you'll learn

- Master Core ADF Components Understand linked services, datasets, pipelines, triggers, and integration runtimes through real-world use cases.
- Build Metadata-Driven Pipelines Automate dynamic data ingestion using Lookup, ForEach, and parameterized datasets.
- Integrate Diverse Data Sources Seamlessly connect to Azure SQL, PostgreSQL, Snowflake, REST APIs, and on-premise systems using SHIR.
- Implement Incremental Loads & CDC Design efficient pipelines using last processed values, tumbling window triggers, and change data capture (Insert, Update, D
- Optimize Performance & Concurrency Tune pipeline execution, manage concurrency, and clean up storage for cost-effective data engineering. - Design Event-Drive
- Design Event-Driven & Scheduled Workflows Trigger pipelines from emails, attachments, and blob events with multi-trigger dependencies.
- Transform Data with Data Flows Apply joins, pivots, unpivots, DQ checks, and Slowly Changing Dimensions (SCD1 & SCD2) using ADF Data Flows.
- Integrate ADF with Databricks & SQL Pools Schedule notebooks, create external tables, and build hybrid architectures for scalable analytics.
- Build Reusable Frameworks & Audit Systems Create centralized logging, reusable ingestion frameworks, and dynamic schema mapping for enterprise-grade solutions

Course content

12 sections • 43 lectures • 43h 40m total length

Azure Data Factory Main Components Copy data between Storage Accounts (Blob)1:16:10
Explore how to copy data between two blob storages using Azure Data Factory, configuring linked services, datasets, and a copy activity powered by integration runtime.

Extract JSON, Parquet Data Using Prefix & Wildcard Filters1:04:53
Explore using Azure Data Factory to copy data from source blob storage to a sync blob, including single and multiple files, JSON and Parquet formats, with prefix and wildcard filters.
Automate Folder Copy with Dataset Parameters1:06:40
Mastering Azure Data Factory teaches automating the copy of multiple folders from source blob to sync blob using dataset parameters, with two linked services and two parameterized datasets.
Metadata-Driven Folder Copy with Lookup & ForEach55:19
Learn to design a metadata-driven pipeline that copies multiple data lake folders to blob storage using lookup and for each, with parameterized datasets and copy activities in Azure Data Factory.

Dynamically Copy Multiple Tables from SQL DB1:19:52
Copy multiple tables from sql database to azure data lake using azure data factory, driven by lookup and for-each with schema and table parameters and active flag filtering.
Ingest Data Using Tables, Queries, & Procedures1:04:35
Learn to ingest data from Azure SQL database to ADLS Gen2 using tables, queries, and stored procedures, with full loads, selective columns, and dynamic pipeline parameters.
Incremental Load with Last Processing Value1:09:11
Explore incremental load in Azure Data Factory by tracking the last processed value, copying new or changed records to data lake, and updating the last processed value after each run.
Optimizing Incremental Loads for Multiple Tables56:56
Explore multi-table incremental loads from Azure SQL Database to data lake using Azure Data Factory, detailing incremental vs full loads, max value tracking, lookups, foreach, copy, and stored procedures.

Install SHIR for On-Prem SQL1:07:04
Install a self-hosted integration runtime to connect on-premise SQL Server with VPN, and design an Azure Data Factory pipeline to copy multiple tables to Azure SQL Database using script activity.
Copy Data to Dedicated SQL Pool1:06:52
Copy data from on-premise sql server to azure dedicated sql pool using bulk insert, polybase, or upsert, orchestrated with a self-hosted integration runtime and adf pipelines.
Transfer Files to Data Lake Storage1:09:13
Learn to copy on premise file system folders to data lake storage gen2 using Azure Data Factory, self-hosted integration runtime, and a configuration-driven for-each pipeline with get metadata checks.
Copy Data from Local SQL Server to Azure SQL1:17:33
Copy data from on-premise sql server to azure sql using azure data factory, employing self-hosted integration runtime, linked services, get metadata, for each, and script activities to handle active tables.

Copy Data from REST API to ADLS Gen259:55
Learn how to copy data from a token-based REST API to Azure Data Lake Storage Gen2 using Azure Data Factory, including token generation, header management, and mapping.
Copy Tables from Snowflake to SQLDB1:16:15
Learn how to copy multiple tables from Snowflake to Azure SQL Database using Azure Data Factory, with staging in blob storage, and configure linked services and pipelines for data ingestion.

Event-Based Triggers & Captured Properties1:03:39
Explore event-based triggers in Azure Data Factory, automating pipelines on file creation or deletion via Event Grid, with capture properties and parameterized datasets for blob to data lake transfers.
Tumbling Window Trigger & Dependencies1:02:31
Master the tumbling window trigger in Azure Data Factory to perform incremental loads from ADLS Gen2, manage trigger outputs and dependencies, and implement windowed data ingestion.
Daily, Weekly & Monthly Schedule Trigger1:03:08
Master Azure data factory schedule triggers to automate pipelines on daily, weekly, and monthly schedules. Learn time zone options, recurrence, start and end dates, and multi-trigger relationships.
Trigger Pipelines from Email Attachments1:02:24
Trigger an Azure Data Factory pipeline when an email with attachments arrives. A Logic App saves attachments to blob storage, and an event-based trigger runs the copy to Azure SQL.
Automate Notifications with Attachments55:55
Automate notifications with attachments after data processing using azure data factory and logic apps. Create event-driven and scheduled workflows that copy data to blob and email summary attachments.
Event-Driven Ingestion Pipelines1:00:53
Explore event-driven ingestion pipelines in Azure Data Factory that start after multiple trigger files are ready in blob storage, using until, get metadata, and execute pipeline activities.
Multi-Trigger Dependencies in Shared Directories1:00:36
Master multi trigger dependencies for ingestion pipelines in shared directories, using get metadata, until, for each, and switch to validate trigger files before executing the data ingestion pipeline.

Copy Data from Multiple Databases55:41
Copy data from multiple tables across databases on the same server into a single Azure SQL database using parameterized linked services, datasets, and a for-each workflow.
Copy Excel Files with Dynamic Schema54:59
Copy excel files from multiple folders into an Azure SQL Database using dynamic schema mappings in Azure Data Factory, with get metadata and for each tracking source and file names.
Consolidate Excel Schemas into SQL Tables55:02
Learn to ingest multiple Excel sources with different schemas into a unified Azure SQL table using Azure Data Factory, with dynamic schema mappings and a config table for per-file mappings.

Optimize Pipeline Concurrency1:03:51
Learn to optimize pipeline concurrency in Azure Data Factory by using a control table to prevent overlapping hourly ingestions, with a master pipeline that checks, starts, and updates run statuses.
Reusable Ingestion Framework1:14:01
Learn to design a reusable ingestion framework in Azure Data Factory that uses a common config table and pipeline parameters to copy data from SQL database and blob to ADLS.
Clean Up Small and Old Files in ADLS Gen258:39
Learn to clean up ADLS Gen2 by deleting small files under a configurable size and aging out files by time, with soft-delete recovery, using Azure Data Factory.

Getting Started with Data Flows1:00:50
Explore mapping data flows in Azure Data Factory to transform data visually within a Spark cluster, covering data ingestion into the data lake, incremental loads, and data quality checks.
SQL Basics & Window Functions1:06:28
Master SQL basics and window functions for data engineering, covering select, join, group by, partition by, order by, where, distinct, and transactions, with row_number, rank, and dense_rank.
Mastering SQL Joins & Types53:25
Explore SQL joins and window functions, compare group by and partition by, and apply inner, left, right, full, and cross joins with practical examples and running totals.
DQ Checks: Duplicates, Nulls, Date Formats43:36
Master data quality checks in Azure Data Factory by identifying duplicates, nulls, and incorrect date formats using data flow, window, and row_number, then separate good and bad records.
Implementing SCD Type 11:25:02
Explore implementing slowly changing dimension type 1 with Azure Data Factory data flows, using CRC 32 hashing to detect changes and handle new, updated, and old records.
Implementing SCD Type 21:01:47
Implement slowly changing dimension type 2 with data flows, using a hash function to detect changes, preserving history with start and end dates, and inserting updates as new records.
SCD Type 1 with ADLS Gen2 Sink50:54
Learn to implement SCD type 1 with an ADLS Gen2 sink using data flows, handling updates and inserts in a data lake via staging and CRC32 hash keys.
Pivot and Unpivot Transformations46:22
Learn pivot and unpivot transformations in Azure Data Factory data flows, grouping by credit id and name, pivoting by card type, handling sums and nulls; orchestrate with pipelines.

Requirements

- No prior experience with Azure Data Factory required All concepts are explained step-by-step through hands-on, real-world projects.
- Basic understanding of data engineering is helpful but not mandatory Topics like ETL, data pipelines, and cloud storage are introduced and reinforced throughout the course.
- Familiarity with SQL is useful While basic SQL knowledge can help, key queries and procedures are demonstrated in the sessions.
- Introductory knowledge of Azure services is a plus Concepts like Azure Storage, Azure SQL, and resource groups are covered as needed.
- Access to an Azure account (free or paid) Recommended for practicing the hands-on labs, but not required to follow along.

Description

Are you tired of theoretical lectures and generic certification prep? This course is built for learners who want to master Azure Data Factory the way it's used in the real world—through hands-on projects, dynamic pipelines, and practical problem-solving.

Led by CloudPandith (Mallaiah), a Senior Data Architect and individual trainer, this course walks you through 40+ real-time ADF scenarios—from ingesting data across SQL, REST APIs, Snowflake, and PostgreSQL, to building reusable frameworks, implementing incremental loads, and transforming data with Data Flows.

Whether you're a beginner or a working professional, you'll learn how to design, automate, and optimize data pipelines using ADF’s full feature set—including SHIR, triggers, Key Vault integration, CDC, and Databricks scheduling. Every session is built around real-world use cases, not textbook theory.

What Makes This Course Different

Real-time projects, not just demos
Metadata-driven and reusable pipeline design
Integration with diverse sources: SQL, REST, Snowflake, Excel, On-Prem
Advanced transformations: SCDs, DQ checks, pivots, joins
Triggering pipelines via events, schedules, and email attachments
Performance optimization and audit logging strategies
Taught by a practicing architect—not a generic instructor

By the end of this course, you’ll be able to confidently build enterprise-grade data pipelines in Azure Data Factory, backed by practical experience and future-ready skills.

Who this course is for:

- Aspiring Data Engineers Beginners looking to break into cloud data engineering with hands-on, project-based learning—no prior ADF experience required.
- Working Professionals & Analysts SQL developers, BI analysts, and ETL practitioners who want to upskill in Azure Data Factory and automate data workflows. - Cloud Engineers & Architects Those building scalable data solutions on Azure and seeking reusable frameworks, metadata-driven pipelines, and performance optimization. - Students & Career Switchers Learners who prefer real-world scenarios over theoretical lectures and want to build job-ready skills with confidence. - Anyone tired of generic training If you're looking for authentic, practical guidance from someone who teaches what they use professionally—this course is for you. Would you like a version of this tailored for your WhatsApp or YouTube audience too? I can make it more conversational and motivational to match your branding.
- Cloud Engineers & Architects Those building scalable data solutions on Azure and seeking reusable frameworks, metadata-driven pipelines, and performance optimization.
- Students & Career Switchers Learners who prefer real-world scenarios over theoretical lectures and want to build job-ready skills with confidence. - Anyone tired of generic training If you're looking for authentic, practical guidance from someone who teaches what they use professionally—this course is for you. Would you like a version of this tailored for your WhatsApp or YouTube audience too? I can make it more conversational and motivational to match your branding.
- Anyone tired of generic training If you're looking for authentic, practical guidance from someone who teaches what they use professionally—this course is for you.

Mastering Azure Data Factory : Real-World Projects With ADF

What you'll learn

Explore related topics

Course content

Introduction to Azure Data Factory1 lecture • 1hr 16min

Blob Storage Projects & Filtering Techniques3 lectures • 3hr 7min

SQL-Based Ingestion Projects4 lectures • 4hr 31min

PostgreSQL Integration Projects1 lecture • 1hr 3min

On-Premise Integration Using SHIR4 lectures • 4hr 41min

External Source Integration (REST API & Snowflake)2 lectures • 2hr 16min

Triggering Pipelines with Events & Schedules7 lectures • 7hr 9min

Multi-Source & Excel-Based Ingestion3 lectures • 2hr 46min

Performance Optimization & Reusability3 lectures • 3hr 17min

Data Transformation with Data Flows8 lectures • 7hr 48min

Requirements

Description

Who this course is for: