Azure Data Factory Training--Continuous Integration/Delivery

Master the basics of Azure Data Factory by copying data from Azure SQL DB to Azure Storage Account using linked services, datasets, and a simple pipeline with a copy activity.

Create an Azure SQL database as the source by configuring a new resource group, a unique server name, and development settings with SQL authentication and a strong password, then deploy.

Connect to an azure sql database via the portal query editor or sql server management studio, adjust firewall rules, and access adventure work data with sql queries.

Create and populate Azure SQL database tables, including customer, address, and sales, insert five sample records, and verify data via query editor or SQL Server Management Studio for copy activity.

Create an Azure storage account in the portal, selecting subscription, resource group, unique name, and location; enable standard locally redundant storage with optional data lake storage Gen2 features.

Download and configure Azure Storage Explorer, sign in to a storage account, and connect via connection string, shared access signature, or account key; upload files or folders to a container.

Create an azure data factory v2 instance through the portal by configuring subscription, resource group, a unique name, and networking, then launch studio to explore monitoring and templates.

Explore the Azure Data Factory interface—manage, monitor, and author pipelines—configure integration runtimes (Azure, self-hosted, SSIS), linked services, triggers, and templates, with git integration.

Create linked services for the Azure SQL database source. Test connections and configure datasets and a copy activity in an Azure Data Factory pipeline.

Learn how to create and configure datasets in Azure Data Factory, including a SQL DB source dataset and a blob storage destination, with linked services, schema import, and file formats.

Create a pipeline in Azure Data Factory and configure a copy data activity from an Azure SQL DB to a CSV delimited sink, with source and sink datasets and mapping.

Execute a data factory pipeline that copies data from Azure SQL DB to Azure Storage, using source and target datasets and a copy activity, then monitor executions.

Delete all resources by removing the resource group after completing a lab, which removes all deployed resources such as database, storage, and data factory; note that this is irreversible.

Copy data from Azure SQL DB to storage and then to the data lake using Azure Data Factory. Build the workflow with linked services, datasets, and pipelines.

Create a new resource group named ADF to act as a container for related resources in East US, review, create, and know that deleting the group removes all resources.

Outline steps to create an Azure storage account, including subscription, resource group, performance, redundancy, data lake with hierarchical namespace, networking, encryption, and final review.

Discover azure data lake storage gen2, a cloud data storage uniting blob storage and hadoop for structured and unstructured data; learn to create accounts, hierarchical containers, and multi-protocol access.

Create a new Azure data lake storage by provisioning a storage account and enabling hierarchical namespace, then configure replication and encryption with a Microsoft managed key for blob and file.

This tutorial walks you through creating an Azure SQL database from scratch—setting up resource group, server, SQL authentication, public endpoint, development config, and adjustable compute and storage.

Connect to the Azure SQL database and configure firewall rules. Create three tables (customer, address, and sales) and insert dummy data to seed five rows per table for source-to-sink copy.

Create an Azure Data Factory instance with basic settings and deploy it. Configure three linked services to access Azure SQL Database, Azure Blob Storage, and Azure Data Lake Storage Gen2.

Create multiple datasets in Azure Data Factory by linking to an Azure SQL DB table, then configure blob storage and data lake delimited text datasets and publish.

Create a multi-activity pipeline in Azure Data Factory by configuring two copy activities: from Azure SQL DB to blob storage, then from storage to Data Lake, using debug.

Learn to clean up Azure Data Factory by deleting the resource group, which removes storage accounts, data factory, and databases, with an irreversible warning and the requirement to specify ADF.

Learn to use parameterized datasets in Azure Data Factory to create dynamic pipelines that copy multiple files with only two datasets and file-name parameters.

Parameterize datasets and pipelines in Azure Data Factory to create a dynamic copy pipeline driven by input and output file name parameters.

Learn how to use pipeline parameters and variables in Azure Data Factory; parameters are constants, while variables update during execution with set variable, and variables are pipeline-scoped.

Learn to access and use system variables in Azure Data Factory, including pipeline name, pipeline run ID, and trigger details, across schedule and event triggers.

Learn to derive file names with expressions in a set variable activity using the concatenate function to form sales_month_year.csv, and parameterize the copy input.

Learn to use annotations to tag pipelines and datasets for easy filtering in Azure Data Factory, and add up to five user properties with dynamic values to track activity changes.

Develop skills to manage files in Azure Data Factory by using get metadata to read file size and name, loop with for each, and delete files under a threshold.

Learn how the get metadata activity in Azure Data Factory extracts metadata from files, folders, and tables, including last modified and size, to guide pipeline decisions.

Learn to use the get metadata activity with azure sql db datasets, examining column count, exist, and structure, and reviewing the customer table's five columns.

Learn to use the wait activity in Azure Data Factory to delay a pipeline by a chosen duration in seconds, for testing or waiting for access.

Master the for each activity in Azure Data Factory to run actions on every item in a list, with sequential or parallel execution and metadata-driven inputs.

Use the if condition activity in Azure Data Factory to branch pipelines by evaluating conditions, linking get metadata output and file size to true or false actions: copy or discard.

Use the delete activity in Azure Data Factory to remove files or folders. Back up data, verify write permissions, and enable logging for safe, auditable deletions.

Practice solving a hands-on Azure Data Factory assignment by deleting files under 150 bytes from storage, configuring if conditions for each activity, and omitting the wait activity.

Solve a practical Azure Data Factory task by using get metadata with a for-each to check file sizes and delete files below a threshold, streamlining CI/CD workflows.

Demonstrates creating a sales details table, inserting sample data across years, and extracting year-based results with Azure Data Factory using stored procedures, lookup, and for each activity.

Demonstrates using an Azure Data Factory lookup activity with an Azure SQL dataset to fetch single or multiple rows, then drive a for-each loop with wait activities per year.

Configure a switch activity in Azure Data Factory to route data by year into three containers, using lookup results and year-based copy activities.

Learn to implement incremental copy activity in Azure Data Factory that transfers only files updated in the last two days from a source to a destination using last modified date.

Explore incremental versus full loads in data pipelines. Learn to use last modified date and a control table to load only new or updated records into the data warehouse.

Implement incremental load in Azure Data Factory using a data table and a control table, with a pipeline that uses two lookups and a stored procedure to update last modified.

Create two Azure SQL databases and a data factory to demonstrate incremental load. Learn resource setup, servers, and deployment steps for tables and stored procedures in the data factory.

Connect to source and destination databases in azure data factory, enable firewall, and create source and destination tables; implement a stored procedure to update last modified date.

Create two linked services to access source and destination SQL databases, create three datasets, and build a pipeline to copy data with full load.

Create a full load and then implement an incremental load in Azure Data Factory, using a copy data activity to transfer only recently updated records from source to destination.

Perform an incremental load in Azure Data Factory using lookup to fetch match date and last modified date, copy new data to a SQL database, and update the control table.

Learn how slowly changing dimension strategies handle changing attributes in data warehouses for Azure Data Factory projects, including type zero, type one, and type two with effective dates.

Implement slowly changing dimension type 1 in Azure Data Factory by comparing source and destination, updating changed records and inserting new ones, via a copy pipeline.

Demonstrates implementing SCD type 1 in Azure Data Factory using a dataflow pipeline with source and destination datasets, inserting new records and updating existing ones.

Learn to configure Azure Data Factory to send custom email notifications via Azure Logic Apps, triggering on pipeline success or failure and orchestrating conditional actions.

Configure a logic app with an http request trigger to send an email via Outlook, generating a URL to invoke the workflow and validating the email notification.

Pass a custom message from Azure Data Factory to Azure Logic Apps using a post request with a JSON payload containing title and message.

Learn to pass system variable values from Azure Data Factory to Logic Apps for email, including data factory name, pipeline name, trigger name, and execution time, via Json updates.

Configure Azure Data Factory and Logic App to send an email when a pipeline fails, using a failure output connected to a web activity and a failed notification email.

Configure Azure Data Factory to send a failed email notification for any activity by using a dedicated execute pipeline that triggers a copy data pipeline to send the alert.

Read data from Azure SQL DB and copy to on-premise storage using Azure Data Factory, with auto resolve and self-hosted integration runtimes, via a pipeline with lookup and for each.

Create a resource group, set up an Azure SQL Database and a server, and create an Azure Data Factory instance for the demo.

Publish two datasets and configure an Azure SQL database, then run a copy activity to export data from Azure SQL to a delimited text file, and use a lookup activity.

Build Azure Data Factory pipelines that leverage a lookup activity to read tables, parameterize datasets, and drive copy activities with dynamic values, including for each to process multiple tables.

Learn to build Azure Data Factory pipelines using a lookup to list table names and a for each activity to copy data for each table with a self-hosted integration runtime.