Learn Azure Data Factory

Name: Learn Azure Data Factory
Rating: 4.3 (24 reviews)

Comprehensive Azure Data Factory Practical Hands-on tutorial

Created byShameer Achary

Last updated 1/2024

English

What you'll learn

You will learn how to build a real-world data pipeline in Azure Data Factory (ADF).
You will learn how to ingest data from sources such as SQL DB, Cosmo DB and Azure Blob Storage into Azure Blob Storage using Azure Data Factory (ADF)
You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into Azure Blob Storage
You will learn about Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines.
You will learn about Integration Runtime , its capability and its types.
How to implement using transformation using Data flow.
You will be able to get a good hand on experience as this course is designed to have more practical session instead of theory.

Course content

13 sections • 82 lectures • 5h 24m total length

Introduction1:29
Course Structure2:56
What is Azure Data Factory?3:55

Module Overview0:28
Overview on what all we will learn in this section
Resource Group Overview1:28
Overview of what is Resource group
Hands on - Resource Group Creation2:02
Hands on -Resource group creation
Hands on -Azure Data Factory Creation4:34
Hands on -Azure Data Factory Creation
Hands on -Storage Account Creation5:21
Hands on -Storage Account Creation
Hands on -Azure SQL DB Creation6:25
Hands on -Azure SQL DB Creation
Hands on -Azure Cosmo DB Creation8:02
Hands on -Azure Cosmo DB Creation
Terminate Resource and Resource Group2:17
How to terminate resource or resource group
Summary of Environment Setup0:38

Module Overview0:24
Overview on what all we will learn in this section
Pipeline And Activity1:35
What is a Pipeline and Activity?
Linked Services And Datasets2:19
What is a Linked Services and Datasets?
Relationship between Pipeline, Dataset, Linked services and activity.
Hands on -Create a Linked Services3:14
Creating linked services for storage account
Supported file format0:48
Supported Source and target file formats
Hands on -Create a Dataset5:58
Creating dataset to files from blob storage account
Triggers0:53
What is trigger and different types of triggers?
Hands on -Create a Pipeline9:58
Create a pipeline.
Publish it and trigger it.
Check the pipeline execution logs, explored pipeline id, activity id, start time of the pipeline.

Copy data from blob storage to adls gen25:23
Copy data from blob storage to adls gen2.
Convert data from csv to parquet format
Copy data from SQL DB to ADLS Gen28:46
Copy data from SQL DB to ADLS Gen2
Rename column in target system2:12
Change the source column name while writing data into target system
Copy data from Azure Cosmo DB for No SQL to ADLS Gen29:02
Copy data from Azure Cosmo DB for No SQL to ADLS Gen2.
Create json format output file in ADLS Gen2.

Introduction to Data Flow1:35
What is data flow and what all are the different parts in Data flow Canvas
Schema drift in mapping data flow1:41
Understand what is schema drift.
What all are the options available in data flow to handle this.
Source and Sink in Data Flows14:40
Derived Column4:12
Select1:51
Filter2:48
New Branch2:48
Apply the filter component to pass only records meeting a condition, using the expression builder to set student ID equals a value. The data preview shows one record after filtering.
Join4:46
Cast2:44
Sort3:32
Conditional split-Scenario 15:11
Conditional split-Scenario 24:22
Conditional split-Scenario 33:08
Lookup3:58
Aggregate6:20
Union2:51
Exist6:05
Learn to implement incremental loads in Azure Data Factory data flow by comparing source orders to a reference output, using the exist transformation to identify new records to migrate.
Surrogate Key2:07
Rank3:28
Alter Row1:51
Flowlet1:43

What is trigger0:52
Schedule Trigger4:55
Storage event trigger4:14
Tumbling Window Trigger Overview3:30
Tumbling Window Trigger (Max concurrency and Delay)3:30
Tumbling Window Trigger (Retry Policy Count and interval in seconds)1:33
Tumbling Window Trigger (Add Dependency)2:25
Tumbling Window Trigger (Offset and Window Size)4:04
Difference between Tumbling window and scheduled trigger1:55
Summary of Triggers0:50

Requirements

Basic understanding about cloud computing.
An Azure Account is required.
Basic understanding of Azure services like resource group, subscription, SQL server, Cosmo DB, Storage account.
Basic understanding of data warehousing concepts (like facts, dimension, database, schema, SCD1, SCD2 etc)
Should be aware of basic SQL query.

Description

Welcome Students.

This course has an in depth technical course for learning Azure Data Factory (ADF). I believe that practical sessions are more effective then the theory session. So this course is designed in such a way that most of the lectures are accompanied with hand on sessions so that students can go through the video and practice side by side.

After covering the technical sessions along with its hands-on, we will go through some real world scenarios and see how those can be implemented using ADF.

The learnings & knowledge gained in this course can be considered as a foundational step in starting your journey towards becoming an Azure Data Engineer.

Currently courses teach you following:-

Azure Data Factory

Overview of what is Azure Data Factory
Create a resource group, Azure SQL DB, Blob Storage, Cosmo DB, Azure Data Factory studio.
Helps us understand what is Linked services, DataSets, Activities, Pipeline, Integration Runtime, Triggers.
How to copy data from source system to target system
What is Data Flow and different components in Data flow.
Different control flow activities.
Parameters and Variables
Monitoring
Explains how to implement real world scenario using ADF.

I welcome you to this course & hope that you gain useful skills and solve complex problems.

Happy Learning!

Who this course is for:

University students looking for a career in Data Engineering
Beginners in Azure Platform
Anyone who is looking forward to start his career as a Azure Data Engineer
Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies
IT developers trying to move to Data Engineering
Data Scientists who want extend their knowledge into data engineering
Data Architects looking to gain an understanding about Azure Data Engineering stack

Learn Azure Data Factory

What you'll learn

Explore related topics

Course content

Introduction3 lectures • 8min

Environment Setup9 lectures • 31min

Building Blocks Of ADF8 lectures • 25min

Integration Runtime2 lectures • 3min

Copy Data Activity4 lectures • 25min

Data Flows21 lectures • 1hr 22min

Parameters & Variables7 lectures • 22min

Connecting multiple activities1 lecture • 6min

Control Flow Activities10 lectures • 28min

Triggers10 lectures • 28min

Requirements

Description

Who this course is for: