[2026] Azure Data Factory : from Zero to Hero of ADF

Create your azure account using a Microsoft account, claim 12 months of popular services and 40 plus free services, plus a $200 credit for 30 days, with phone verification.

Explore the Azure portal by logging in, using the search bar, and navigating resources, cloud shell, and subscriptions; manage resource groups, costs, and access controls.

Create and configure an Azure storage account with subscription, resource group, and region; enable secure transfer, block public access, select private container, and access via account keys and endpoints.

Explore Azure storage account redundancy options, from locally redundant storage to zone and geo redundancies, and learn how data replicates across data centers and region pairs to withstand failures.

Create and configure an Azure data lake storage Gen2 account by enabling the hierarchical namespace, selecting subscription, resource group, unique account name, region, networking, encryption, and accessing endpoints and keys.

Create an Azure data factory resource, configure a basic setup, and explore the data factory studio across author, monitor, and manage tabs for pipelines and datasets.

Create an Azure SQL database by provisioning a new server with a basic 2 GB tier, configuring public access and IP whitelisting, and connecting with Azure Data Studio for queries.

Create a Windows virtual machine on Azure with SQL Server preinstalled to serve as an on-premise demo database; then connect via RDP, create a database, and import CSV data.

Explore the building blocks of Azure Data Factory—pipelines, activities, datasets, linked services, and triggers—and understand how these components connect within a data factory.

Define pipelines as a logical grouping of activities executed sequentially, such as copying data from storage to blob and logging ingestion details to SQL via stored procedure, using project folders.

Explore Azure Data Factory activities as atomic steps in a pipeline, including move and transform, copy data, dependencies, and conditional flows with on success, on failure, or skipped outcomes.

Linked services are the connection strings Azure Data Factory uses to access resources, created in the Azure portal under manage, selecting data stores or compute resources, and testing the connection.

Learn how data sets identify data locations in a data store using a link service and CSV format, and how to create, configure, and connect datasets to a copy activity.

Explore how triggers start pipelines, how activities run with datasets and link services, and how connection strings connect data stores in the Azure Data Factory component hierarchy.

Enable logging in settings to store copy activity details in Azure Blob, selecting a connection and container with info or warning levels and modes like best effort or reliable.

Learn to copy multiple csv files from a folder with the copy data activity in Azure Data Factory, using folder datasets, recursive inclusion, and wildcard filters to the output container.

Demonstrates adding a dynamic file name column during copy from blob storage to a SQL table, creating a five-column destination and validating with a debug run.

Copy data from Azure SQL tables to Azure Intelligent Storage using Azure Data Factory, creating a link service, datasets, and a copy activity.

Demonstrate the fault-tolerant copy data activity by automatically discarding incompatible rows, inserting only compatible data to the sync, and logging exact reasons to a blob storage folder.

Set up the self-hosted integration runtime to bridge on-premise storage and databases with Azure Data Factory, installing, registering, and configuring authentication keys to enable data copy workflows.

Understand variables in Azure Data Factory, how they differ from parameters, and create string, boolean, and array variables with defaults, edit them using set variable during execution.

Use the set variable activity to assign a value to a pipeline variable, selecting the variable and using a pipeline parameter with dynamic content like the current timestamp.

Learn to use the filter activity in an Azure Data Factory pipeline to filter array elements by a condition, using the item keyword with a greater-than expression.

Discover how the lookup activity reads datasets from data sources in Azure Data Factory, returns a singleton or array to drive for each or filter steps, and its 5000-row limit.

Copy in bulk all tables from a database to ADLs using a lookup to fetch tables, then loop with for-each and copy each.

Master retrieving data store metadata with the get metadata activity in Azure Data Factory, using outputs like item name, item type, size, and structure in pipelines.

Learn how the if condition activity in Azure Data Factory uses a boolean parameter to branch execution, copying input data to India or US folders via copy activities.

Use the wait activity in an Azure Data Factory pipeline to pause for a time specified by dynamic content or a lookup activity, 15 seconds, with user properties for validation.

Learn to use the stored procedure activity in Azure Data Factory to create and execute parameterized SQL procedures, pass arguments, and run delete or other statements within a pipeline.

Log Azure Data Factory pipeline run details into a SQL table for audit by using a stored procedure and system variables to capture pipeline name and trigger type.

Learn to use the fail activity in Azure Data Factory to deliberately fail a pipeline when data don’t match expectations, set error codes and messages, and alert stakeholders.

Use the execute pipeline activity to invoke child pipelines from a parent pipeline, set to wait for completion, publish changes, and test by debugging two child pipelines.

Use dataflow activity in pipelines to transform and move data with mapping data flows, enabling derived columns, sorts, group by, and aggregates on spark clusters.

Learn mapping data flows in Azure Data Factory, a visual, code-free way to design data transformations that run as pipeline activities on auto-provisioned Spark clusters, with data preview and debugging.

Configure the source transformation in Azure Data Factory data flows, validate the schema, and manage schema drift with inferred types and default formats.

Configure sync transformation to store transformed data, map input to output, log error rows, and include update methods, key columns, table actions, pre and post scripts, and continue on error.

Learn how the conditional split in Azure Data Factory routes data into streams using matching conditions, with a default case for other employees.

Learn how to use the union transformation in Azure Data Factory to merge three streams—employee, HR, and others—into a single data set and preview the merged results.

Apply rank transformation in Azure Data Factory to create a rank column based on salary in descending order, using normal rank and dense rank, with ties affecting skipped ranks.

Combine two data sources in a mapping dataflow with the join transformation. Join on department ID, use left outer joins, and preview results to see nulls for non matches.

Learn how the pivot transformation in Azure Data Factory converts gender values into department-level columns using group by and aggregates, producing total male and total female columns.

Understand how lookup transformations reference data from a secondary source in your data flow, behaving like a left outer join by enriching the primary stream with matching rows and nulls.

Explore derived column transformation in Azure Data Factory to generate new columns and modify existing ones, such as creating salary in USD and handling missing department names.

Flatten JSON data by unrolling arrays into individual records using the flatten transformation in Azure Data Factory's dataflow, demonstrating how to handle complex fields like employee name and skills.

Demonstrate alter transformation in Azure Data Factory to insert, delete, update, and upset rows, using filtered subsets from a CSV source to populate a SQL table.

Learn data quality and validation in Azure Data Factory data flows using expected true checks, unique checks for keys, and exist checks, with derived columns flagging bad rows.

Explore Azure Data Factory expression language and functions to create dynamic pipelines, convert parameters, and build dynamic queries with runtime evaluation and timestamped outputs.

Apply dynamic queries in Azure Data Factory to copy data using a config file, filtering by gender and selecting specific columns for each destination dataset.

Learn how triggers automate Azure Data Factory pipelines, control when they run, and enable one-to-many or many-to-one relationships, with four types: schedule, tumbling window, storage events, and custom events.

Explore the four types of Azure Data Factory triggers: schedule, tumbling window, storage event based, and custom event based triggers, and focus on the first three with a hands-on demo.

Explore how to configure Azure Data Factory schedule triggers to kick off pipelines on a clock-based recurrence, with calendar options, time zones, and start vs scheduled times.

Explore how Azure Data Factory leverages storage event triggers and event grid to invoke pipelines when blob storage files arrive or are deleted.