Complete Azure data factory with projects and scenario

Use Azure Data Factory to build and manage pipelines that extract, transform, and load data from multiple sources into a unified destination.

Learn to create an Azure Data Factory in the portal, configure subscription and resource group, and launch the Azure Data Factory Studio to build pipelines, datasets, and data flows.

Discover Azure Data Factory components—activities, datasets, link services, integration runtimes, and triggers—guided by a blob-to-sql copy scenario.

Explore Azure storage accounts and learn to use blob, file, table, queue storage for unstructured, shared, structured, transactional data, with general purpose v2 and hot, cold, or archive tiers.

Learn to copy data with an Azure Data Factory pipeline, moving data from a blob storage input container to an output container using a copy activity.

Perform a delete activity in Azure Data Factory to remove source data after daily copy from the input container to the output container, with blob storage data set and logging.

Set up an Azure SQL database in portal.azure.com with subscription, resource group, server, and authentication.

Copy CSV data from Azure blob storage into Azure SQL database using an Azure Data Factory pipeline, defining source, sink, datasets, and mapping, then debug and monitor the run.

Explore triggers in Azure Data Factory, including schedule, tumbling window, event, and manual triggers, and learn how each enables automatic or manual pipeline execution based on time, data, or events.

Configure a schedule trigger in Azure Data Factory to automatically run a pipeline at defined intervals, with start time, time zone, and end date.

Understand how event based triggers in Azure Data Factory automatically run pipelines in response to storage events, such as blob created, to copy CSV data to JSON.

Master tumbling window triggers in Azure Data Factory, learn how they schedule pipelines over fixed windows and backfill historical data, and implement a SQL-to-CSV pipeline with stored procedure parameters.

Learn how Azure Data Factory's data flow performs ETL transformations with filter and sort operations, moving data from Azure SQL database to blob storage, including data preview, debugging, and partitioning.

Master the join transformation in Azure Data Factory to combine employee and department CSV files from blob storage. Build a data flow and pipeline to output the employee department dataset.

Learn to use the union transformation in Azure Data Factory to combine Mumbai and Delhi CSV files row-wise into output, and distinguish union from join by row-wise versus columnar merging.

Master the select transformation in Azure Data Factory to rename, drop, and reorder columns, demonstrated with a practical data flow from csv input to csv output.

Explore how the derived column transformation in Azure Data Factory creates or updates columns, replaces nulls with unknown, uppercases address data into a country column, all within a data flow.

Discover how the aggregate transformation in Azure Data Factory performs sum, mean, max, and count with group by, demonstrated on a csv data flow and salary analysis.

Explore how parameterization in Azure Data Factory passes runtime values to a single dataset, enabling copying multiple SQL tables to blob storage without creating hundreds of datasets.

Learn to parameterize a single Azure Data Factory link service to connect to multiple SQL databases and servers by passing db name, username, and password at runtime.

Instantiate a single data factory pipeline that parameterizes dataset and container names to copy multiple sql tables into their respective blob storage containers at runtime.

Learn how to save files in azure data factory with a proper naming convention and json format using a copy activity, and convert time from utc to indian standard time.

Discover how the get metadata activity in Azure Data Factory reveals metadata like item name, item type, last modified, and exist to filter and copy specific CSV files.

Explore how to use the for each activity in Azure Data Factory to iterate multiple files, read metadata, and copy each CSV file to JSON in another storage container.

Demonstrates using the if condition activity in Azure Data Factory to route files by name, leveraging get metadata and for each item copy EMP files to employee container.

Learn to use the set variable activity in azure data factory to create and assign string, boolean, array, and integer values, concatenate variables to form file names for copy data.

Learn how to use the append variable activity in Azure Data Factory to append values to an array and verify results with a set variable in a pipeline.

This lecture explains the stored procedure activity in Azure Data Factory, showing how to call a stored procedure from a pipeline, pass parameters, and delete a record by id.

Learn how the lookup activity in Azure Data Factory reads a table or file and returns results, then uses for each loop and a stored procedure to delete resigned employees.

Learn how the wait activity in Azure Data Factory pauses a pipeline for a specified time between copy data steps, enabling delays such as a 30-minute interval before loading data.

Introduce the switch activity in Azure Data Factory, using expressions and parameters to route copy data to employee or department folders based on a folder name.

Explore how the validation activity in Azure Data Factory verifies a dataset reference exists at runtime, guiding success to copy data or failure to email notification.

Explore how to use the execute pipeline activity in Azure Data Factory to invoke another pipeline, with options for wait on completion and parallel execution.

Learn to apply a filter activity in Azure Data Factory to select file names from get metadata output and copy the filtered data with a subsequent copy activity.

Understand how the until activity in Azure Data Factory mimics a do while loop, looping until a file exists in blob storage, using Get Metadata, If Condition, and Set Variable.

Learn how to use web activity in Azure Data Factory to call rest api via pipelines, pass json bodies, and handle get and post requests with datasets and link services.

Understand how to use the fail activity in Azure Data Factory to throw errors under certain conditions, via a switch-based example that copies data for employee one or two.

Learn to use the script activity in Azure Data Factory to run DML and DDL against Azure SQL, Synapse, SQL Server, Oracle, and Snowflake, with parameters and stored procedures.

Explore the exist transformation in Azure Data Factory data flows to filter records by exist and not exist checks across left and right streams using blob storage csv sources.

Explore the conditional split transformation in Azure Data Factory data flows, routing data by location (India, US, UK) into separate output containers using case-like conditions.

Apply the pivot transformation in Azure Data Factory data flows to convert unique country values into columns, grouping by department and counting employees per country, then store results as csv.

Learn how the unpivot transformation reverses pivot by turning column values into rows in an Azure Data Factory data flow, with practical steps from setup to output.

Learn how to apply the surrogate key transformation in Azure Data Factory to add an incremental id per row, with start and step values, for dimension tables.

Learn how to use the windows transformation in Azure Data Factory to perform ranking and cumulative sums, with over clause, partition by, and order by.

Master the alter row transformation in azure data factory by applying insert, update, delete, and upsert on row-level data within a data flow, using conditions to filter or modify records.

Explore rank and dense rank transformations in Azure Data Factory data flow to rank salaries in descending order, creating a new rank column and handling data type conversion.

Resolve the Azure SQL Database v2 not supported in a data flow error in Azure Data Factory. Create a data set and switch to a legacy SQL Database link service.

Discover how to fix the import schema failed error when using Azure SQL database as a data set by setting import schema to none and then previewing the table.

Troubleshoot the dataflow schema import failed in Azure Data Factory by resetting the schema, importing the projection, and validating with data preview before storing to blob storage via a pipeline.

Demonstrate an end-to-end Azure Data Factory project implementing SCD type 1, using CSV data from Blob storage to upsert into Azure SQL by employee ID.

Create an end-to-end Azure Data Factory pipeline that copies only non-empty csv files from input to output blob containers, using get metadata, for each, set variable, lookup, and if condition.

Build an end-to-end Azure Data Factory workflow that deletes files older than seven days from a blob storage input container using get metadata, for each, and delete activities.

Migrate data from on prem sql server to Azure Storage using Azure Data Factory, with a self-hosted integration runtime, building a pipeline and copying table data to blob storage.

Build an end-to-end Azure Data Factory pipeline implementing SCD type two to insert new records and update existing ones in blob storage csv data, using hash and is active flag.

Validate incoming files in an Azure Data Factory pipeline by using get metadata to compare the file structure with a predefined schema, and process only when the structures match.

Create an Azure data factory pipeline to copy csv files from an input blob container to json files in an output container using a csv dataset and a wildcard path.