Azure Data Factory Workshop: Real-World Project Use Cases

Explore the fundamentals of cloud computing and why we move from on-premise data centers to cloud solutions. Learn public, private, hybrid clouds, IaaS, PaaS, SaaS, plus Azure and pay-as-you-go.

Learn to create an Azure SQL Database as a platform-as-a-service school database, manage a resource group, and use Data Factory to move data between the database and data lake storage.

Create an Azure Data Lake Storage Gen2 account, understand data lake concepts, and learn to store raw, schema-free data at scale for staging and later analysis.

Create an Azure SQL virtual machine with Windows and SQL Server license, configure networking, and deploy; then connect via remote desktop and create a project database.

Explore the design of a cloud-based data warehouse built with Azure Data Factory. Trace end-to-end data flow from files and a SQL Server VM to staging and the Azure database.

Learn how Azure data factory moves on-premise data to the cloud by extracting, transforming, and loading through pipelines that connect data sources, apply cleansing, publish, schedule, and monitor.

Discover the building blocks of Azure Data Factory, including pipelines, activities, datasets, linked services, triggers, and data flows, and their integration with runtimes.

Explore loading data into a staging area with Azure Data Factory, using a self-hosted runtime to copy from a VM SQL Server and flat file, plus full vs incremental loads.

Load multiple staging tables from multiple sources using a single dynamic pipeline. Implement incremental data loading with a watermark and conflict table.

Create dimensions and a fact table in a star schema using Azure Data Factory workflows, linking customer, product category, and store dimensions to a central fact for real-world reporting.

Explore the concept of slowly changing dimension type 1, updating existing customer records in place without history, using staging, delta loads, and data flow in Azure Data Factory.

Define a loading strategy for dimensions and fact in a data warehouse, choosing slowly changing vs incremental loading, with customer city changes, static product categories, and insert-only fact design.

Explore building an Azure data factory pipeline to load a dim customer table from staging, with type one slowly changing dimension logic, dataflow transformations, and debugging previews.

Load product category and subcategory dimension incrementally by inserting only new records from staging, using dataset mappings and existence checks. Validate pipeline updates by confirming the dimension records count.

This pipeline extracts unique store types from the transactions file using an aggregate, then loads only new stores into the store dimension via a daily lookup.

Load fact transactions into the data warehouse using a data factory pipeline and data flow, performing left outer joins of staging with customer and product category dimensions.

Create a master pipeline in Azure Data Factory to orchestrate and sequence staging, dimension, and fact pipelines. Schedule, monitor, and publish the workflow for enterprise-level data orchestration.