Complete Azure data factory with projects and scenario

Use Azure Data Factory to build and manage pipelines that extract, transform, and load data from multiple sources into a unified destination.

Learn to create an Azure Data Factory in the portal, configure subscription and resource group, and launch the Azure Data Factory Studio to build pipelines, datasets, and data flows.

Explore Azure storage accounts and learn to use blob, file, table, queue storage for unstructured, shared, structured, transactional data, with general purpose v2 and hot, cold, or archive tiers.

Learn to copy data with an Azure Data Factory pipeline, moving data from a blob storage input container to an output container using a copy activity.

Set up an Azure SQL database in portal.azure.com with subscription, resource group, server, and authentication.

Copy CSV data from Azure blob storage into Azure SQL database using an Azure Data Factory pipeline, defining source, sink, datasets, and mapping, then debug and monitor the run.

Explore triggers in Azure Data Factory, including schedule, tumbling window, event, and manual triggers, and learn how each enables automatic or manual pipeline execution based on time, data, or events.

Configure a schedule trigger in Azure Data Factory to automatically run a pipeline at defined intervals, with start time, time zone, and end date.

Understand how event based triggers in Azure Data Factory automatically run pipelines in response to storage events, such as blob created, to copy CSV data to JSON.

Master tumbling window triggers in Azure Data Factory, learn how they schedule pipelines over fixed windows and backfill historical data, and implement a SQL-to-CSV pipeline with stored procedure parameters.

Master the join transformation in Azure Data Factory to combine employee and department CSV files from blob storage. Build a data flow and pipeline to output the employee department dataset.

Master the select transformation in Azure Data Factory to rename, drop, and reorder columns, demonstrated with a practical data flow from csv input to csv output.

Explore how the derived column transformation in Azure Data Factory creates or updates columns, replaces nulls with unknown, uppercases address data into a country column, all within a data flow.

Discover how the aggregate transformation in Azure Data Factory performs sum, mean, max, and count with group by, demonstrated on a csv data flow and salary analysis.

Explore how parameterization in Azure Data Factory passes runtime values to a single dataset, enabling copying multiple SQL tables to blob storage without creating hundreds of datasets.

Learn to parameterize a single Azure Data Factory link service to connect to multiple SQL databases and servers by passing db name, username, and password at runtime.

Instantiate a single data factory pipeline that parameterizes dataset and container names to copy multiple sql tables into their respective blob storage containers at runtime.

Discover how the get metadata activity in Azure Data Factory reveals metadata like item name, item type, last modified, and exist to filter and copy specific CSV files.

Explore how to use the for each activity in Azure Data Factory to iterate multiple files, read metadata, and copy each CSV file to JSON in another storage container.

Demonstrates using the if condition activity in Azure Data Factory to route files by name, leveraging get metadata and for each item copy EMP files to employee container.

Learn to use the set variable activity in azure data factory to create and assign string, boolean, array, and integer values, concatenate variables to form file names for copy data.

This lecture explains the stored procedure activity in Azure Data Factory, showing how to call a stored procedure from a pipeline, pass parameters, and delete a record by id.

Learn how the lookup activity in Azure Data Factory reads a table or file and returns results, then uses for each loop and a stored procedure to delete resigned employees.

Learn how the wait activity in Azure Data Factory pauses a pipeline for a specified time between copy data steps, enabling delays such as a 30-minute interval before loading data.

Introduce the switch activity in Azure Data Factory, using expressions and parameters to route copy data to employee or department folders based on a folder name.

Explore how the validation activity in Azure Data Factory verifies a dataset reference exists at runtime, guiding success to copy data or failure to email notification.

Explore how to use the execute pipeline activity in Azure Data Factory to invoke another pipeline, with options for wait on completion and parallel execution.

Learn to apply a filter activity in Azure Data Factory to select file names from get metadata output and copy the filtered data with a subsequent copy activity.

Understand how the until activity in Azure Data Factory mimics a do while loop, looping until a file exists in blob storage, using Get Metadata, If Condition, and Set Variable.

Learn how to use web activity in Azure Data Factory to call rest api via pipelines, pass json bodies, and handle get and post requests with datasets and link services.

Understand how to use the fail activity in Azure Data Factory to throw errors under certain conditions, via a switch-based example that copies data for employee one or two.

Explore the exist transformation in Azure Data Factory data flows to filter records by exist and not exist checks across left and right streams using blob storage csv sources.

Explore the conditional split transformation in Azure Data Factory data flows, routing data by location (India, US, UK) into separate output containers using case-like conditions.

Apply the pivot transformation in Azure Data Factory data flows to convert unique country values into columns, grouping by department and counting employees per country, then store results as csv.

Learn how to apply the surrogate key transformation in Azure Data Factory to add an incremental id per row, with start and step values, for dimension tables.

Learn how to use the windows transformation in Azure Data Factory to perform ranking and cumulative sums, with over clause, partition by, and order by.

Troubleshoot the dataflow schema import failed in Azure Data Factory by resetting the schema, importing the projection, and validating with data preview before storing to blob storage via a pipeline.

Demonstrate an end-to-end Azure Data Factory project implementing SCD type 1, using CSV data from Blob storage to upsert into Azure SQL by employee ID.

Create an end-to-end Azure Data Factory pipeline that copies only non-empty csv files from input to output blob containers, using get metadata, for each, set variable, lookup, and if condition.

Build an end-to-end Azure Data Factory workflow that deletes files older than seven days from a blob storage input container using get metadata, for each, and delete activities.

Migrate data from on prem sql server to Azure Storage using Azure Data Factory, with a self-hosted integration runtime, building a pipeline and copying table data to blob storage.

Build an end-to-end Azure Data Factory pipeline implementing SCD type two to insert new records and update existing ones in blob storage csv data, using hash and is active flag.

Create an Azure data factory pipeline to copy csv files from an input blob container to json files in an output container using a csv dataset and a wildcard path.