Masterclass on Azure Data Factory - Data Engineer for 2026

Explore copy activity in Azure Data Factory to move data between on-premises and cloud stores, configuring source and sink, supporting diverse data stores, file formats, and post-move transformation and analysis.

Learn how delete activity in Azure Data Factory removes files or folders in blob storage to clean up archives, using dataset options like file path, wildcards, prefixes, and recursion.

Invoke another pipeline from a master pipeline using the execute pipeline activity in Azure Data Factory, demonstrating master and child pipelines, lookup, and dataset integration.

Explore how the get metadata activity retrieves file, folder, or table metadata in Azure Data Factory or Synapse pipelines, and how to use output in conditional expressions and downstream activities.

Learn how the lookup activity fetches data from a file or table in Azure Data Factory, returning configuration data or query results for use in subsequent activities.

Explore set variable in Azure Data Factory to declare and initialize variables, then compute area from length and breadth with dynamic expressions, int conversion, mul, and string formatting.

Understand how the wait activity in Azure Data Factory holds a pipeline for a specified time, using seconds (60) or 3600 seconds to manage dependencies from external sources.

Explore how the filter activity in Azure Data Factory applies a condition to each array item, using contains to filter substrings and output an array of matching values.

Explain how the until activity loops until its condition becomes true, using a variable, incrementing it, and performing type conversions with set variable and add operations.

Demonstrate how the Azure Data Factory if condition evaluates a boolean to execute true or false branches, using variables and set variable to determine the interest rate.

Apply join transformations in Azure Data Factory data flows to merge two sources using inner, left, right, and full outer joins, handling matched and unmatched records with nulls.

Aggregate transformation in Azure Data Factory performs group-by style aggregations on columns to compute average salary, max salary, and counts per department.

Learn the select transformation in Azure Data Factory data flows, a SQL-like tool to rename, drop, and reorder columns without altering row data.

Learn how to derive a new column or modify existing fields in Azure Data Factory data flows, using the derived column transformation and storing results to blob storage.

Master the pivot transformation in Azure Data Factory data flows, turning unique values from a single column into multiple columns by grouping by department and counting by job.

Explore unpivot transformation in Azure Data Factory to convert a wide dataset into a vertical view by ungrouping a column and turning column names into a values column.

Explore the rank transformation in Azure Data Factory data flows, generating dense rankings from a sorted salary column in ascending or descending order and exporting results to a file.

Explore the flatten transformation in Azure Data Factory data flows to unroll array values in JSON into individual rows, demonstrating data normalization and outputting flattened results to a blob destination.

Learn how the filter transformation in Azure Data Factory data flows filters rows by a condition, like a SQL where clause, producing only matching records and/or supporting multiple conditions.

Copy files dynamically from a blob storage container to a destination using Azure Data Factory, designing linked services, datasets, and a get metadata plus loop to move all files.

Explore how the set variable activity assigns values to pipeline variables. Learn to increment with a temporary variable and int conversion to avoid self assignment.

Design an Azure Data Factory pipeline to migrate files whose names contain a substring pattern from source blob storage to a destination, using get metadata, filter, and copy activities.

Log pipeline details after execution using a stored procedure into an ADF_pipeline_log table, recording data factory name, pipeline name, run ID, and timestamp.

Split a semicolon-delimited CSV line into product, description, and area columns using Azure Data Factory Data Flows; derive columns, map to SQL Server destination, and validate pipeline execution.

Learn to copy missed files from source to destination in Azure Data Factory by identifying missing files and orchestrating a pipeline with get metadata, filter, and copy activities.

Implement end-to-end email notifications in Azure Data Factory using one stored procedure to manage start, success, and failure, with dynamic subject, body, and recipients.

Discover what a data warehouse is: a central, well-structured repository for cleansed data used for analysis and reporting, with data organized into fact and dimension tables.

Master data warehouse keys by exploring candidate keys, primary keys, foreign keys, and surrogate keys, and learn how each key functions in data design.

Identify a candidate key as a column that is both unique and not null, as shown by employee id or passport; names may repeat, so they are not candidate keys.

Discover how a foreign key links an employee to a department by referencing the dept table's primary key, using dept number as the reference, and see self-referential keys like mgr.

Define surrogate key as a system generated, auto-incremented unique identifier that serves as the primary key, especially in slowly changing dimensions for historical tracking.

Explore dimension and fact tables in a star schema, including primary and surrogate keys, descriptive attributes, and measurable facts; learn how incremental and full loads impact the data model.

Learn how to implement SCD type 2 in Azure Data Factory to preserve full history with surrogate keys, start and end dates, and incremental loads via staging and merge scripts.

Learn the scd type 4 approach with a current dimension table and a history table, using staging data and a surrogate key to track changes and auditing.

Understand SCD type 5, using a mini dimension to track city changes while keeping current values in the main dimension for fast queries.

Implement SCD type 6 by maintaining history and the current value in a single record with dates and active flag. Learn to merge staging data and insert new history records.

Create a logs table under the afm schema with an identity id and fields for adf name, pipeline name, run id, start time, status, and error messages, illustrating identity usage.

Create two stored procedures to log Azure Data Factory pipeline starts and ends, inserting data factory name, pipeline name, run id, and start/end times with dynamic values and testing.

Create an Azure Key Vault to securely store secrets like passwords and access keys, with current and older versions. Apply them in Data Factory pipelines and Databricks mounting.