Azure Data Factory Essentials Training (Hands-On)

Learn to create and configure budgets in cost management for your subscription, set monthly thresholds, define scope, and receive email notifications as spending approaches limits.

Create your azure data factory instance from the portal or PowerShell, configuring subscription, resource group, region, and networking, then review and monitor pipelines and data flows.

Learn to create an Azure Data Factory with PowerShell in Cloud Shell, defining the resource group, location, and factory name. Verify the new factory in the portal.

Understand how a pipeline groups activities to ingest, clean, and analyze data with a mapping data flow. Learn to create, name, and set concurrency for pipelines in Azure Data Factory.

Learn to move and transform data with Azure Data Factory components, including data movement and transformation activities, datasets, linked services, and pipelines, plus control flows like if conditions and switches.

Learn to build dynamic Azure Data Factory pipelines by using parameters to pass values to datasets, share data between activities, and apply global parameters for scalable pipelines.

Pass parameters between activities, extract file metadata with a get metadata activity, and populate a SQL Server table via a stored procedure.

Learn to use global parameters in Azure Data Factory to share values across pipelines and activities, reference them with fully qualified names, and verify reuse in demos.

Explore the four trigger types in Azure Data Factory: skiddle, tumbling, storage event, and custom events, and learn how to attach pipelines, set schedules, and manage retries.

Explore Azure integration runtime types—Azure, self-hosted, and ssis—demonstrating how the default, serverless Azure IR powers real-time data integration across Azure and external clouds, with managed cost and network considerations.

Install and validate a self-hosted integration runtime on your own compute using authentication keys, then monitor connectivity, diagnostics, and resource-based scaling for more concurrent jobs.

Link a self-hosted integration runtime to an existing data factory to reuse a single runtime. Configure permissions and sharing to establish a one-to-one relationship and manage keys across factories.

Explore Azure-SSIS integration runtime to lift and shift on-premises SSIS packages to a fully managed cloud, using Bring Your Own License and region-aware deployment.

Ingest data from a local Excel file into Azure Data Lake Gen2 using copy activity. Configure linked services, datasets, and a pipeline with a self-hosted integration runtime for end-to-end movement.

Learn to copy data from AWS S3 to SQL Server using Azure Data Factory by configuring a linked service, uploading a parquet file, testing connections, and creating the copy activity.

Create an Azure Data Factory linked service for Azure SQL Database using auto resolve integration runtime, then configure managed identity authentication and database access permissions via Management Studio.

Ingest parquet files from s3 into azure sql database using azure data factory by creating datasets and linked services, then build a copy pipeline that auto-creates the target table.

Learn to monitor Azure Data Factory pipelines by triggering runs, viewing executions, and analyzing throughput, records, and consumption to ensure successful data transfers.

Explore multiple inputs and outputs transformations in azure data factory, including join types (left, right, full, cross), conditional split, exists, union, and lookup to append columns from matched data.

Flatten JSON by unrolling nested nodes into rows to simplify handling hierarchical data. Use formatters to parse delimited data and XML, and to format strings efficiently.

Explore the row modifier transformation, enabling conditional search, update, and delete operations on rows based on prioritized expressions, mirroring a merge-like approach to database changes.

Explore how pipelines use sinks as destinations to insert data, and work with visual transformations that run behind the scenes, optimized automatically on cluster environments.

Explore defining a source in mapping data flows by choosing between datasets and inline sources, comparing connectors and performance, and using Azure SQL DB as the source.

Configure source options in Azure Data Factory by setting the task name and enabling drift. Auto-detect new columns with drifted column types, enforce validated schema, and use sampling for debugging.

Spin up a data flow spark cluster by selecting the integration runtime and configuring time to live, then wait for initialization and confirm readiness to ingest data.

Define the data schema by using projection to set column types, infer types from sample text files, and manage predefined schemas from databases, with caution when overwriting.

Explore partitioning options in Azure Data Factory Essentials Training, including six types and green partitioning, to optimize loads with predefined rules and appropriate partition keys.

View a sample schema and data types, then use data preview to fetch and inspect table data and the columns from left and right tables as you apply transformations.

Create and configure a sink in a mapping data flow by creating a new dataset and table, mapping columns one-to-one, enabling error handling, then publish and run the pipeline.

Execute a mapping data flow by running it within a pipeline, configuring runtime, logging, and validation. Publish, trigger, and monitor the pipeline to ingest and transfer data.

Explore how to ingest, transform, and aggregate crime data for metropolitan cities using Azure Data Factory in a project walkthrough that covers storage accounts, containers, and workspace setup.

Create an Azure Databricks workspace, configure region and pricing, and import notebooks from GitHub; explore data ingestion notebooks and establish connectivity with Azure Data Factory.

Build a Databricks and Azure Data Factory pipeline to ingest data, configure blob storage linked services with managed identity, test connections, and run the data transfer.

Validate data transfer in a Databricks and Data Factory workflow by verifying the copy task outputs in the sink dataset, inspecting blob storage, and running notebook experiments to transform data.

Learn to orchestrate data transformation by linking an Azure Data Factory notebook activity to a Databricks notebook, publish and run pipelines, monitor results, and validate transformed data.

Create an azure devops organization and a private project, naming the organization near your region. Navigate the left menu to manage repositories and pipelines within the project.

Create a new Git repository in Azure DevOps, initialize it by adding a readme file, note the default branch, and connect your data factory to this repository.

Link data factory to an Azure DevOps repository by configuring a code repository, choosing the collaboration and publisher branches, and importing existing objects to the root.

Learn how to version Azure Data Factory with branches, manage development to production releases via repositories and pull requests, and deploy pipelines and data flows through a release pipeline.

Merge data factory code from a feature branch into the main branch using a pull request, approve the merge, and publish to trigger pipelines.

Learn to build a CICD pipeline for Azure Data Factory in Azure DevOps by creating release pipelines, selecting artifacts, configuring environments, and validating deployments with incremental options.

Learn how to execute a release pipeline in Azure DevOps for ADF, including enabling continuous deployment triggers and pre-deployment approvals, and deploy to production data factories with verification.