Azure Data Factory | Data Engineering on Azure and Fabric

Explore Azure storage options, including blob storage for unstructured data with hot, cool, and archive tiers and lifecycle policies, and data lake storage Gen2 built on blob storage for analytics.

Classify data into structured, semi-structured, and unstructured, detailing fixed schemas, relational tables, and metadata-driven formats like json. Explore how sql querying, management, and tradeoffs differ across these data types.

Explore Parquet, a columnar, type-aware file format that preserves column data types via metadata and supports compression with snappy, GZIP and lasso for efficient large-scale data processing.

Download and unzip the customer orders dataset, review json, bar k, and text delimited files, and understand the fact table relationships to orders, customers, stores, and products for Azure storage.

Outline the data lake blueprint by loading raw data into a landing zone, moving to raw, cleansed, and structured zones, enriching and converting to the K format for analytics.

Create a storage account and data lake in Azure, choosing subscription, resource group, and redundancy, then enable hierarchical namespace for data lake gen two and prepare containers for landing files.

Navigate the storage interface, create a landing container, upload orders, and use soft delete to recover blobs; create data lake containers raw, cleansed, structured, analytics for Azure Data Factory workflows.

Explore how Azure Data Factory processes diverse data with a code-free platform for data engineers, enabling hybrid ETL, ELT, and data integration across cloud and on-premises stores.

Create and configure an Azure data factory resource in the portal, selecting resource group, subscription, region, and v2, then launch the data factory studio to access overview and activity log.

Navigate the azure data factory interface to author activities, including copy activity, and pipelines. Define source and sync data sets, link services, manage integration runtimes, and monitor with dashboards.

Master the copy activity, a data movement operation that copies data from a source to a destination within a pipeline, enabling integration, backups, migrations, and warehousing.

Define the section objective: copy data unchanged from the landing container in Azure Blob Storage to the raw container in the Data Lake Gen2 storage account using a copy activity.

Configure Azure Data Factory linked services for blob storage and Data Lake Gen2, create source and sink datasets for copy activity from landing to raw containers, and publish after validation.

Create a pipeline and configure a copy activity to move order items from the landing to the raw container in the data lake, then validate and publish.

Configure a copy activity in Azure Data Factory to move the customers.json from the landing container to the rule container in data lake gen2, using source and sink json datasets.

Create and run an Azure Data Factory pipeline to copy orders parquet from landing to rule, configuring source and sink datasets, a copy activity, and compression with snappy.

Create json data sets for products and stores, then build copy pipelines to move files from the landing to the raw container in the data lake Gentoo account.

Organize data factory resources by creating folders for landing and raw datasets, consolidate copy data activities into a single pipeline, then validate, debug, and publish changes.

Chain activities in Azure Data Factory pipelines to create dependencies, control execution with on success or on failure, and debug data movement from landing to raw across copy data activities.

Explore data factory copy activity behaviors for multiple JSON files using wildcard paths, recursion, and options to merge, preserve, or flatten hierarchies.

Explore mapping data flows in Azure Data Factory, a visual, code-free way to transform data loaded into Spark data frames on managed clusters, with validation, enrichment, reusability, and monitoring.

Use data flows to move and cleanse data from raw to cleansed to structured and analytics containers in lake gen two storage, delivering datasets for analysts and data scientists.

Demonstrate building your first data flow in Azure Data Factory, from a source to a delimited text sink, using the data factory studio canvas with debugging and pipeline execution.

Learn how dataflow debug tests and visualizes transformations in data flows, with per-step data previews, and understand auto resolve integration runtime pricing and turning off debugging to save costs.

Understand how partitioning splits data into partitions for parallel Spark execution, and how source and sink partitioning shape processing and output, including key-based, round-robin, hash, and dynamic range methods.

Explore the select transformation in data flows to rename, drop, and reorder columns without changing rows, enabling downstream column control through a demo data flow DF underscore select.

Use the cast transformation in Azure Data Factory dataflow to change column data types and surface casting errors, with a sample DF_cast and data preview.

Importing the projection in data flows defines the data shape with columns and data types, and is recommended to enable validation and transforms like filtering and pivoting.

Demonstrates the derived column transformation in a dataflow by creating and modifying columns with expressions, such as lowercasing status, concatenating values, and converting strings to timestamps using to timestamp.

Create a single data flow to transform data from the raw container into cleansed parquet files in the cleansed container, deriving order_date from order_date_time and adding updated_timestamp with current UTC.

Create a raw to cleansed data flow in Azure Data Factory, using parquet sinks and datasets for orders, order items, customers, products, and stores, with derived updated timestamps and preview.

Learn to read partitioned datasets by using a root folder and parquet files to combine partitions with the same schema, and use partition route path to reveal order_status.

Apply filter and sort transformations in Azure Data Factory data flows to filter by order status (refund, cancelled) or date, and sort by unit price descending.

Explore aggregate transformations in data flows to compute sums, minimum, maximum, and counts by grouping by order ID and product ID, generating total order quantities and minimum unit prices.

Explore how to join data streams with inner, left, right, and outer joins on common keys such as store id or department id to produce a unified dataset.

Explore conditional split and union transformations in Azure Data Factory to route orders data into complete, refunded, and cancelled streams and then recombine them into a single dataset.

Build a single data flow in Azure Data Factory to transform cleansed container data into the structured container, join orders with order_items, and derive subtotals from unit price and quantity.

Learn to design a cleansed to structured data flow in Azure Data Factory, creating and syncing orders, products, and stores datasets, applying joins, derived columns, and updated timestamps.

Derive month and year from order_date, filter out cancelled orders, and aggregate product and store monthly sales in Azure Data Factory data flows for analytics.

Create data sets for product orders monthly and stores monthly, then build a data flow that joins, filters, derives month year, and aggregates totals for analytics csv outputs.

Learn to orchestrate Azure Data Factory pipelines by using the execute pipeline activity to run multiple pipelines sequentially, with dependencies, wait on completion, and monitor outputs.

Schedule triggers in Azure Data Factory enable attaching a recurring daily, weekly, or monthly pipeline execution at a specific time, with start date, time zone, and monitoring.

Learn to create storage events triggers in Azure Data Factory, monitor blob creation with Event Grid, and run the pipeline with a copy activity from the landing to raw containers.

Prepare a data lake by creating source and sink folders in a test container, upload three orders files for Jan 29–31, and configure delimited text datasets in Azure Data Factory.

Learn how to use the delete activity in Azure Data Factory to remove files or folders after a pipeline run, including logging for audit purposes.

Use the get metadata activity to retrieve folder and file metadata in data factory pipelines, enabling conditional expressions with outputs like exists, child items, name, type, and last modified.

Learn how the lookup activity retrieves data from supported sources, configure a pipeline, and toggle between returning the first row or all rows in json for demonstration purposes.

Explore how if condition and switch activities control Azure Data Factory pipelines by evaluating file existence or column counts, using get metadata outputs to trigger copy activities.

Pass external values into pipelines, datasets, and data flows with parameters in Azure Data Factory. Create and use dataset and pipeline parameters with dynamic content to parameterize file names.

Explore Azure Data Factory variables, creating and setting values in a pipeline using set variable, including string and array types, with dynamic content via UTC and format date time.

Explore the for each activity that iterates over a collection to execute a loop of activities. See examples using arrays and get metadata to process file names sequentially.

Apply a metadata driven copy in azure data factory using a single source and sink dataset to move csv files to a date-stamped sink with dynamic listing.

Implement a daily data flow moving landing files to raw, cleansed, and structured containers with date-based folders, partition orders by order date, and overwrite products and stores; note Upsa limitations.

Learn to use azure data factory to move and ingest data into and as your sql database, using sql databases as sources and syncs for copy activities and data flows.

Create an Azure SQL database and server, set a SQL admin login, enable a public endpoint, and choose a basic DTU model. Review compute, storage, and the query editor.

Create a new schema orders and three tables—orders, products, stores—in SQL, define columns and types, and establish a composite key order_item_id from order_id and product_id.

Create a unique identifier for orders by concatenating order id and product id in a data flow, converting to strings, then back to integer, and rearranging columns before publishing.

Create an Azure SQL linked service and three datasets for orders, products, and stores using the integration runtime, test the connection, import schemas, and publish the changes.

Dynamic datasets automatically update their file paths to today’s date in raw and cleansed folders, and you must re-run pipelines after clearing containers to avoid data duplication.

Demonstrate copying data from a structured dataset to an SQL sink in azure data factory, including pre copy scripts, schema mapping, and upsert behavior.

Learn to use the alter row transformation in a data flow with SQL sync to apply insert, delete, update, and observe policies, including conditional upsert handling and key column mapping.

Modify the data flow to route sinks to SQL tables, apply an alter row transformation with a universal policy for orders, and truncate stores and products before full loads.

Tumbling window triggers fire at fixed intervals, running pipelines within a defined window with start time, end time, and window size, enabling scheduling, retries, and trigger dependencies.

In this lecture, use tumbling window triggers and pipeline parameters to ingest data only between specified window start and end times, via a stored procedure and copy activity.

Learn to create tumbling window trigger dependencies between two pipelines, so one runs only after the other completes successfully, using built-in trigger dependencies.

Explore authentication options for Azure Data Factory, including account key, service principals, and managed identities, and learn how to securely manage keys, access control, and rotation.

Learn how to use Azure Key Vault to store secrets such as account keys, configure access policies, create secrets, and link to Azure Data Factory for secure credential management.

Learn to configure Azure Data Factory with system assigned managed identities to authenticate a linked service to a data lake, assigning blob data contributor RBAC roles at the resource level.

Create a user assigned managed identity, attach it to your data factory, grant storage blob data contributor access, and configure a link service to the data lake using the credential.

Create an app registration and a service principal with storage blob data contributor access to your data lake, then configure Azure Data Factory linked service authenticated by this service principal.

Apply fine-grained access control to Azure storage by assigning blob data contributor roles to a managed identity at container level, enabling linked services for data factory without storage account access.