Udemy Business

Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Azure Data Factory - Data Engineering With Real Time Project

Name: Azure Data Factory - Data Engineering With Real Time Project
Rating: 4.6 (48 reviews)

Real Time Project Using Azure Data Factory , ADLS Gen2 , Azure SQL Database , Key Vault , Logic Apps and Azure Monitor

Created byRagunathan Ramanujam

Last updated 9/2025

English

What you'll learn

Design and Development of Azure Data Factory Pipelines for Data Ingestion
Design and Development of Azure Data Factory Pipelines for Data Transformation & Loading
Design and Development of Azure Data Factory Pipelines for Data Analytical Reporting Database
Design and Development of Azure Data Factory to process Semi Structured Data

Course content

16 sections • 175 lectures • 15h 37m total length

Data Analytics And Data Engineering Introduction16:10
Azure Data Analytical Services Introduction19:02
Azure Cloud Fundamentals Introduction5:58
Free Azure Account for Students1:33
Free Azure Account for All4:43

Data Ingestion Pipeline Requirement5:47
Pipeline Expression Builder & Dataset Parameters Overview3:58
Creating & Configuring Dataset Parameters7:07
Pipeline Parameters Overview6:46
Creating & Configuring Pipeline Parameters5:53
Debugging & Dynamically Passing Parameter Values6:32
Recap Dataset and Pipeline Parameters7:02
Identify Static and Dynamic Parameter Values5:20
Configure Static Pipeline Parameter Values8:12
Dynamic Pipeline Expressions Using Pipeline Parameters & ADF Functions9:06
Data factory DATETIME format overview & Error Handling in Ingestion Pipeline6:48
Formatting DATETIME Values Using Advanced Expressions and Date Functions6:09
Configuring Pipeline Variables and SET VARIABLE Activity7:03
ADF Triggers Overview6:07
Create & Configure Tumbling Window Trigger6:17
Passing Tumbling Window Trigger System Variables To Pipeline Parameters5:30
Monitor Automated Pipeline Runs Via Tumbling Window Trigger4:27

Metadata Driven Ingestion Pipeline Overview6:20
Metadata Driven Ingestion Pipeline Requirement5:10
Create Metadata File6:15
Create a metadata JSON file to manage two source file name patterns for an Azure Data Factory pipeline, replacing inline parameters and enabling reading metadata within ADF.
Upload Metadata File and LOOKUP Activity Overview5:41
Configure LOOKUP Activity - Read Metadata File4:32
Configure LOOKUP Activity Settings & Read Metadata File Values as Array6:55
Configure FOR EACH Activity to Loop Through LOOKUP Output Value Array6:23
Configure FOREACH Activity To Pass Metadata Values to Dataset Parameters8:19
Debugging Metadata Driven Ingestion Pipeline & Error Handling5:59
Organising Folder Structure in ADF4:47
End To End Metadata Driven Ingestion Pipeline - Create Metadata File5:47
End To End Metadata Driven Ingestion Pipeline - Create Datasets5:57
End To End Metadata Driven Ingestion Pipeline - Create Pipeline Part 17:04
End To End Metadata Driven Ingestion Pipeline - Create Pipeline Part 28:54
End To End Metadata Driven Ingestion Pipeline - Debugging And Error Handling4:00
End To End Metadata Driven Ingestion Pipeline - Create Tumbling Window Trigger7:55
Data Ingestion From HTTP Web Server - Summary2:17

Dimensional Data Modelling Overview4:23
Dimensional Data Modelling - Source Data Overview2:50
Identify Business Description(Dimension) and Business Measures(Fact) Columns5:54
Identify Business Description(Dimension) Column Groups Having 1-1 Relationship5:16
Logical Data Modelling - Designing Dimension Tables3:53
Create dimension tables from grouped business description columns, assign surrogate keys (state_id, market_id, product_id, variety_id), and prepare the corresponding fact table.
Logical Data Modelling - Designing Fact Tables4:41
Physical Data Modelling Overview2:42
Physical Data Modelling - Identifying Data Types6:21
Physical Data Modelling - Applying Naming Conventions1:57
Physical Data Modelling - Including Audit Columns and Define Database Schema3:40
Setup AZURE SQL DATABASE - Overview0:46
Azure SQL Database Query Editor Overview6:20
SQL Server Management Studio Overview2:03

Dimension Table Load Overview0:56
Dimension Table Load - Data Transformations Overview5:47
Dimension Table Load - Configure Source Dataset4:14
Dimension Table Load - Configure SQL Database Linked Services and Sink Dataset3:35
Dimension Table Load - COPYDATA Limitations and Usage Requirement for DATAFLOW4:58
Dimension Table Load - DATAFLOW Overview6:13
Dimension Table Load - DATAFLOW SELECT Transformation4:26
Dimension Table Load - DATAFLOW AGGREGATE Transformation5:45
Dimension Table Load - DATAFLOW EXPRESSION BUILDER Overview5:40
Dimension Table Load - DATAFLOW SURROGATE KEY Transformation3:42
Dimension Table Load - DATAFLOW DERIVED COLUMN and SINK Transformation5:41
Dimension Table Load - Recap DATAFLOW Transformations & Debugging DATAFLOW4:57
Dimension Table Load - Change Data Capture (CDC) Overview5:33
Dimension Table Load - Change Data Capture Add Sink Data Into Source Stream5:55
Enable change data capture for the dim state table by using separate source and sync datasets, renaming columns, truncating the target table, and preparing to merge streams.
Dimension Table Load - Change Data Capture LOOKUP Transformation5:29
Dimension Table Load - Change Data Capture FILTER Transformation6:07
Dimension Table Load - Change Data Capture SURROGATE KEY Limitation5:59
Dimension Table Load - Change Data Capture Get Max SURROGATE KEY Using Query6:04
Dimension Table Load - Merge Max SK with Source Using JOINER Transformation5:34
Dimension Table Load - Merge Max SK JOINER Transformation Custom Join Condition6:31
Apply a custom join condition in Azure Data Factory to load a dimension table by merging max state IDs with the surrogate key generator transformation, using derived columns.

Reusable DATAFLOW Design - Create Source and Sink Datasets8:15
Reusable DATAFLOW Design - Configure Source Transformation6:06
Reusable DATAFLOW Design - Generate Dynamic SQL Query Using Dataflow Parameter6:20
Reusable DATAFLOW Design - Dynamic SQL Query Import Projection6:34
Reusable DATAFLOW Design - Dynamic SQL Query For Max Surrogate Key Value5:21
Reusable DATAFLOW Design - Configure SELECT , AGGREGATOR and LOOKUP4:39
Reusable DATAFLOW Design - Configure FILTER , JOINER and DERIVED COLUMN6:09
Reusable DATAFLOW Design - Configure SINK & Pipeline To Automate Run5:13
Reusable DATAFLOW Design - Debugging2:05
Reusable DATAFLOW Design -Produce Dimension and Fact Tables CREATE SQL Scripts2:48
Reusable DATAFLOW Design - Reusing to Populate DIM_MARKET Dimension Table5:41
Reusable DATAFLOW Design - Reusing to Populate DIM_PRODUCT Dimension Table6:08
Reusable DATAFLOW Design - Reusing to Populate DIM_VARIETY Dimension Table6:40
Reusable DATAFLOW Design - Debugging Run Of Loading All Dimension Tables3:46

Incremental Load Overview2:11
Incremental Load - GET METADATA Activity Overview5:20
Incremental Load - GET METADATA Activity Filter Metadata Between Dates6:46
Incremental Load - IF CONDITION Activity Overview5:18
Incremental Load - IF CONDITION Activity For EXCEPTION Handling2:59
Incremental Load - ADF Single Pipeline Limitations & Workaround(Multi Pipeline)3:13
Incremental Load - Calling and Passing Values Between Parameters6:32
Incremental Load - Designing Reporting Tables Load Master Pipeline6:29
Incremental Load - Calling Reporting Dimension Load From Master Pipeline5:13
Incremental Load - Using Master Pipeline GET METADATA output in Child Pipeline4:49
Incremental Load - Debugging Dimension Tables Load Part 16:09
Incremental Load - Debugging Dimension Tables Load Part 23:55
Incremental Load - Configure Tumbling Window Trigger5:27
Incremental Load - Test Tumbling Window Trigger Run1:11
Incremental Load - Fixing Data Quality Error on DIM_VARIETY Table Load1:05
Loading DIM_DATE Dimension Table - Using INSERT STATEMENTS0:10
Data Ingestion Pipeline - HTTP Source Full Load Using Tumbling Window Trigger2:52

Fact Table Load Transformation Overview5:51
Fact Table Load DATAFLOW - Configure Source File2:23
Fact Table Load DATAFLOW - Configure Dimension Table Source & LOOKUP SurrogateID4:58
Fact Table Load DATAFLOW - Configure All Dimension Tables & LOOKUP SurrogateIDs5:53
Configure data flow to load the fact table by cloning and preparing dimension sources (market, product, variety), set up lookups for surrogate IDs, and apply selective projections and naming conventions.
Fact Table Load DATAFLOW - Configure DATE Dimension Table & LOOKUP SurrogateID4:53
Fact Table Load DATAFLOW Debugging and Fixing Data Issues3:49
Fact Table Load DATAFLOW - Configure SINK Transformation5:44
Fact Table Load DATAFLOW Testing Part 17:26
Fact Table Load DATAFLOW - Add Tumbling Window Trigger6:17
Fact Table Load DATAFLOW - Testing End to End Reporting Database Load2:46

Requirements

None

Description

The course includes most of the Real Time Scenarios on Developing Data Analytical Project.

Creating and Configuring Azure Data Lake Storage Gen2, Azure Data Factory, Azure SQL Database

Developing Metadata Driven Ingestion Pipeline to load source data into the Azure Data Lake Storage Account

Expertise knowledge on Lookup , Foreach and Copy activities for Ingesting the Source Data

Dimensional Modelling Design To Design Dimension and Fact Tables

Creating the Dimension Tables in Azure SQL Database

Design and Developing the Azure Data Factory pipelines to load Type1 Dimension Tables

Design and Development of Type2 Dimension Tables load using Azure Data Factory Pipeline

Design and Development of Fact tables using Azure Data Factory Pipeline

Performing Incremental Load in Data Ingestion and Data Transformation Azure Data Factory Pipeline to Process Delta data

Processing Semi Structured data using Azure Data Factory Dataflow Components

Automate the Azure Data Factory pipeline run using different type of Triggers Schedule , Tumbling Window and Event

Configuring Security between Azure Data Factory, Azure Data Lake Storage Account , Azure SQL Database and Azure Key Vault

Using Azure Key Vault to store the Secrets and use the Secrets in ADF pipelines

Monitoring the ADF pipeline runs using ADF Monitor and Azure Monitor

Debugging and Error Handling of Azure Data Factory Pipelines

Who this course is for:

Switch Over to Data Analytics Jobs, Students , Datawarehouse & Business Intelligence Developers, Non Cloud Software Developers

Azure Data Factory - Data Engineering With Real Time Project

What you'll learn

Explore related topics

Course content

Introduction5 lectures • 47min

Azure Portal Overview and Create Data Analytical Resources5 lectures • 31min

Azure Data Factory Introduction8 lectures • 48min

Data Ingestion Pipeline - HTTP Source17 lectures • 1hr 48min

Metadata Driven Data Ingestion Pipeline - HTTP Source17 lectures • 1hr 42min

Reporting Database Design - Dimensional Data Modelling ( Star Schema)13 lectures • 51min

Dimension Tables Load - ADF DATAFLOW Overview20 lectures • 1hr 43min

Reusable DATAFLOW Design For Loading Dimension Tables14 lectures • 1hr 16min

Incremental Load ( Delta Records Processing ) - Dimension Tables Load17 lectures • 1hr 10min

Fact Table Load DATAFLOW Design10 lectures • 50min

Requirements

Description

Who this course is for: