[2026] Mastering Azure Data Factory :From Basics to Advanced

Explore what Azure Data Factory is and how it performs extract, transform, and load to move data from on-premises sources to cloud storage and analytics services.

Create an Azure Data Factory service in the portal, configure subscription, resource group, region, and name, then explore the data factory studio to author, monitor, and manage pipelines.

Explore the building blocks of data factory, including pipelines, link services, datasets, and activities, and learn about triggers and integration runtimes for cloud and self-hosted setups.

Create a hello world data factory pipeline to copy a local file to a data lake Gen2 container using a copy data activity, with source dataset and sink configurations.

Copy data activity scenario 2 demonstrates copying data from Azure storage to Azure SQL database using Azure Data Factory. It covers dataset, linked service, mapping, and a copy pipeline.

Explore the lookup activity, which reads dataset contents from files or tables and returns output; use queries and first row only, noting the 5000-record limit.

Explore how the get metadata activity retrieves metadata from a dataset, showing file and folder details, table schemas, and properties like size, last modified, and child items.

Configure the filter activity to filter the lookup output in Azure Data Factory, using the items input, dynamic content, and the equals function on item fields like product subcategory.

Learn how the for each loop in Azure Data Factory iterates over a set of records using previous activity input, with sequential or parallel execution and inside-loop activities.

Explore how to implement an if condition inside a for-each loop in Azure Data Factory, using the current item reference and equals to branch actions for each lookup result.

Execute pipeline activity triggers other pipelines in a master-child workflow. Select target pipelines from a dropdown, with wait on completion, to invoke a child pipeline from a master pipeline.

Explore the data flow activity in Azure Data Factory, a code-free way to perform data transformations on Spark clusters, using drag-and-drop design, configurable integration runtimes, and reusable dataflows.

Configure the source transformation in Azure Data Factory data flow by adding datasets or inline sources via linked services, then join employer and department data and preview results.

Master the join transformation in a data flow by connecting employee and department sources, configuring left or right joins on joining keys, and outputting employee name, salary, and department.

Use a select transformation to trim a joined data flow to three fields—employee name, salary, and department name—renaming them for clarity.

Apply the exists transformation to compare two inputs, returning left-stream records that exist in the right stream or that do not exist. It outputs only left-stream columns.

Explore the derived column transformation in Azure Data Factory data flows, updating existing columns, creating calculated ones, concatenating names, and using local variables for reusable expressions and validation.

Explore the filter transformation in Azure Data Factory data flows: select an input stream, build an expression, and filter records by salary greater than 5000; preview outputs and validate results.

Identify department-wise maximum salary and total salary using the aggregate transformation after joining employee and department tables; group by department name and apply max and sum functions.

Explore the conditional split transformation to route input data into parallel streams based on split conditions, with first matching condition vs all matching conditions and a default stream.

Create a new branch to implement parallelism in a data flow, splitting output into parallel streams. Apply transformations in a branch, such as join, to process data in parallel.

Drive data loading with dataflow sync transformations, loading transformed outputs to data lake and relational targets, and apply updates using alter clauses and key columns.

Explore how to parameterize pipelines, datasets, and linked services in Azure Data Factory, using dynamic content and runtime evaluation to build reusable, scalable data workflows.

Explore how to monitor the performance of your Azure Data Factory visually, track pipeline and activity executions without writing code, view run details and errors, and set alerts with metrics.

Learn to monitor data factory performance with Azure Monitor and Azure Data Factory Analytics, configure diagnostic settings, and analyze logs and metrics via Log Analytics workspaces and dashboards.

Master incremental data loads with a single Azure Data Factory pipeline that updates multiple staging tables daily using the last updated field and a dynamic enablement flag.

Build an Azure Data Factory pipeline for incremental data load, syncing customer and lead data from the CRM source to destination staging with max last updated tracking.

Learn about slowly changing dimension and how type I handles updates without history, and how incremental load inserts or updates records to maintain or discard history.

Demonstrates implementing slowly changing dimension type 1 in Azure Data Factory data flow by comparing staging and dimension tables, updating existing records, and inserting new ones.

Implement slowly changing dimension type 2 by preserving customer history through staging to dim tables, using a current flag and start/end dates to distinguish latest from historical records.

demonstrates a dataflow pipeline implementing slowly changing dimension type 2. uses a staging table and dimension to detect changes, update current flag, and insert new records to preserve history.