Azure Data Factory Training--Continuous Integration/Delivery

Create an Azure storage account in the portal, selecting subscription, resource group, unique name, and location; enable standard locally redundant storage with optional data lake storage Gen2 features.

Download and configure Azure Storage Explorer, sign in to a storage account, and connect via connection string, shared access signature, or account key; upload files or folders to a container.

Create an azure data factory v2 instance through the portal by configuring subscription, resource group, a unique name, and networking, then launch studio to explore monitoring and templates.

Explore the Azure Data Factory interface—manage, monitor, and author pipelines—configure integration runtimes (Azure, self-hosted, SSIS), linked services, triggers, and templates, with git integration.

Delete all resources by removing the resource group after completing a lab, which removes all deployed resources such as database, storage, and data factory; note that this is irreversible.

Outline steps to create an Azure storage account, including subscription, resource group, performance, redundancy, data lake with hierarchical namespace, networking, encryption, and final review.

Discover azure data lake storage gen2, a cloud data storage uniting blob storage and hadoop for structured and unstructured data; learn to create accounts, hierarchical containers, and multi-protocol access.

This tutorial walks you through creating an Azure SQL database from scratch—setting up resource group, server, SQL authentication, public endpoint, development config, and adjustable compute and storage.

Connect to the Azure SQL database and configure firewall rules. Create three tables (customer, address, and sales) and insert dummy data to seed five rows per table for source-to-sink copy.

Create multiple datasets in Azure Data Factory by linking to an Azure SQL DB table, then configure blob storage and data lake delimited text datasets and publish.

Create a multi-activity pipeline in Azure Data Factory by configuring two copy activities: from Azure SQL DB to blob storage, then from storage to Data Lake, using debug.

Learn to clean up Azure Data Factory by deleting the resource group, which removes storage accounts, data factory, and databases, with an irreversible warning and the requirement to specify ADF.

Learn to use parameterized datasets in Azure Data Factory to create dynamic pipelines that copy multiple files with only two datasets and file-name parameters.

Learn how to use pipeline parameters and variables in Azure Data Factory; parameters are constants, while variables update during execution with set variable, and variables are pipeline-scoped.

Learn to access and use system variables in Azure Data Factory, including pipeline name, pipeline run ID, and trigger details, across schedule and event triggers.

Learn to use annotations to tag pipelines and datasets for easy filtering in Azure Data Factory, and add up to five user properties with dynamic values to track activity changes.

Learn to use the get metadata activity with azure sql db datasets, examining column count, exist, and structure, and reviewing the customer table's five columns.

Use the delete activity in Azure Data Factory to remove files or folders. Back up data, verify write permissions, and enable logging for safe, auditable deletions.

Demonstrates creating a sales details table, inserting sample data across years, and extracting year-based results with Azure Data Factory using stored procedures, lookup, and for each activity.

Demonstrates using an Azure Data Factory lookup activity with an Azure SQL dataset to fetch single or multiple rows, then drive a for-each loop with wait activities per year.

Configure a switch activity in Azure Data Factory to route data by year into three containers, using lookup results and year-based copy activities.

Learn to implement incremental copy activity in Azure Data Factory that transfers only files updated in the last two days from a source to a destination using last modified date.

Implement incremental load in Azure Data Factory using a data table and a control table, with a pipeline that uses two lookups and a stored procedure to update last modified.

Create two Azure SQL databases and a data factory to demonstrate incremental load. Learn resource setup, servers, and deployment steps for tables and stored procedures in the data factory.

Connect to source and destination databases in azure data factory, enable firewall, and create source and destination tables; implement a stored procedure to update last modified date.

Perform an incremental load in Azure Data Factory using lookup to fetch match date and last modified date, copy new data to a SQL database, and update the control table.

Implement slowly changing dimension type 1 in Azure Data Factory by comparing source and destination, updating changed records and inserting new ones, via a copy pipeline.

Read data from Azure SQL DB and copy to on-premise storage using Azure Data Factory, with auto resolve and self-hosted integration runtimes, via a pipeline with lookup and for each.

Publish two datasets and configure an Azure SQL database, then run a copy activity to export data from Azure SQL to a delimited text file, and use a lookup activity.

Learn to build Azure Data Factory pipelines using a lookup to list table names and a for each activity to copy data for each table with a self-hosted integration runtime.