Ultimate Azure Data Factory: Cloud Data Engineering

Explore Azure Data Factory, a cloud-based data integration service, and learn its usefulness and benefits while setting up your Azure account and familiarizing with the Azure Portal interface and terminology.

Explore why Azure Data Factory matters for cloud migration, minimal-code data integration, and orchestrating batch and real-time data with big data, machine learning, and continuous integration and DevOps tools.

Explore Azure Data Factory, a managed cloud service for serverless data integration, transformation, and orchestration. See how its platform-as-a-service model provides pay-as-you-go scalability with no upfront costs and cloud-only processing.

Explore how Azure Data Factory enables data integration across 100 sources and formats, supports code-free transformations, orchestration of pipelines with external compute; includes version control, monitoring, and legacy SSIS compatibility.

Set up your Azure account by choosing pay-as-you-go or free options, claim $200 starter credit, and explore free services like Azure Data Factory while managing usage.

Navigate the Azure portal user interface and terminology, including subscriptions, resource groups, and resources. Create and manage storage accounts, databases, and analytics services like Azure Data Factory and Azure Databricks.

Discover why Azure Data Factory serves as a cloud based ETL and data integration service. Set up an Azure account, access the portal, and create a resource group and resource.

Explore the business scenario and hands-on project for Azure Data Factory, uncover the project requirements, architectural patterns for analytic solutions, and the code and data repositories used in the course.

Explore a three-store wine retailer business case, detailing sales data integration challenges across formats and stores. Build insights on best sellers, store contributions, and growth opportunities while planning data consolidation.

Build a cloud-based, no-code analytics solution that integrates data from multiple sources and is extensible for needs, using Azure Data Factory, Azure SQL DB, data bricks, and Power BI dashboards.

Present an Azure analytics pattern with ingest, store and process, enrich, and serve layers, using Data Factory, Azure Event Hubs, Data Lake Gen2, Delta Lake, Azure Databricks, and Power BI.

Design an Azure data platform using Azure Data Factory to orchestrate a medallion architecture—landing, raw, cleansed, and staging—feeding a star-schema warehouse and Delta Lake via Databricks, with Power BI insights.

Explore the git repository for the ultimate Azure Data Factory course, featuring sample data, delta files, product and master data, notebooks, and ADF pipelines.

Gain a concise view of business challenges, solution requirements, and a typical Azure-based architecture for a modern analytics solution, plus repositories with code and data and data warehousing references.

Create an Azure account and set up Visual Studio Code with SQL, Python, and Databricks extensions, plus GitHub or Azure DevOps, and explore Azure Data Studio and Azure Storage Explorer.

Install Visual Studio Code, configure git, and set up an Azure DevOps account. Link Visual Studio Code to Azure DevOps to manage repositories, commits, and deployments.

Explore the required azure resources for this course, including azure data lake storage gen 2, azure data factory, and azure sql db for staging and the data warehouse.

Explore how to create Azure resources using the Azure portal, Visual Studio Code extensions, or command-line tools, and implement repeatable infrastructure with ARM templates, Bicep, Terraform, or JSON templates.

Create and configure Azure resources for your data project, including a resource group, Azure Data Lake Storage Account Gen2, Data Factory pipelines, and an Azure SQL DB data warehouse.

Create a data lake storage Gen two resource to enable hierarchical folders and role based access control, then configure the storage account and enable hierarchical namespace for Azure data lake.

Create an Azure Data Factory resource within your resource group, select your subscription and region, and use the project-environment-ADF naming with v2 before launching the Data Factory Studio.

Create an Azure SQL db resource from the resource group as a cloud-native SQL database. Set up a new SQL server with SQL authentication for development.

Manage Azure Data Factory, data lake storage Gen2, and Azure SQL resources in the portal, configure access and firewall rules, and prepare to connect with Azure Data Studio.

install azure data studio and connect to azure sql db with a copied server name. browse tables, write queries, and create a table using a desktop, user-friendly interface.

Install and sign in to Azure Storage Explorer, connect to the data lake storage Gen2, create a raw container, and upload files to manage storage from the desktop.

Explore essential tools and workflows for cloud data engineering by configuring Visual Studio Code with extensions, linking to Azure DevOps, and provisioning resources via the Azure portal and ARM templates.

Explore how pipelines group activities to execute tasks in Azure Data Factory, with data movement, data transformation, and data control activities, and trigger pipelines with schedules.

Explore how Azure Data Factory components connect from sources to destinations using linked services and datasets, then execute copy and transformation activities within pipelines on schedules or events.

Navigate the Azure Data Factory user interface in the Azure portal, launch the Azure Data Factory Studio, and access ingest data, create pipelines, and transform data.

Explore the ingest stage of ETL by moving on-prem data to the cloud, storing it in Azure Data Lake Storage Gen2, and transforming it with Azure Data Factory.

Build a one-time Azure Data Factory pipeline that copies and unzips zip files from ADLS Gen2 to a raw folder using binary copy and zip deflate.

Adopt naming conventions for Azure resources and Data Factory components to improve maintainability and readability across resource groups, storage accounts, adls gen two, linked services, pipelines, datasets, and data flows.

Explore activity dependencies in data factory by building a pipeline with a validation activity using the abs csv cleansed sales dataset, then demonstrate success, failure, completion, and skip dependencies.

Learn how the copy data activity infers schema, converts data types through a three-step process, and uses debug mode with previews and data integration units to boost performance.

Learn how the copy data activity infers schemas, configures source-to-sink mappings, and applies type conversion; adjust data integration units and parallelism while debugging pipeline runs.

Explore how expressions and variables enable dynamic Data Factory pipelines by determining runtime properties with the expression builder, using system variables and string interpolation for readable, maintainable code.

Explore expressions and variables in azure data factory by using dynamic content, the concat function, and system variables to create dynamic, reusable pipelines.

Explore runtime parameters in Azure Data Factory to create reusable pipelines by sharing linked services, data sets, and pipeline parameters with optional defaults and global parameters.

Explore how to create and use pipeline and global parameters in Azure Data Factory, pass parameters to variables at runtime, and construct dynamic expressions with concat, balancing reusability and readability.

Create an Azure Key Vault instance, store secrets and encryption keys, and retrieve credentials at runtime to secure linked services and centralize access with Azure Active Directory and logging.

Set up an Azure Key Vault in your resource group, add secrets, and grant Data Factory access through a vault access policy using the managed identity.

Build a data factory pipeline to import semi-structured json files, use get metadata and for each copy to csv in the cleansed sales data folder, using Azure Key Vault.

Explore pipeline activities and dependencies, apply expressions, variables, and parameters, and integrate with Azure Key Vault to build efficient data pipelines using for each and metadata activities.

Discover mapping data flows in Azure Data Factory: visually design and transform data with filtering, joining, and pivoting using an expression builder, debug, and monitor Apache Spark-powered pipelines.

Explore common data engineering use cases for mapping data flows and learn how built-in transformations like derived column, aggregate, conditional split, join, and lookup enable data harmonization and deduplication.

Explore the data factory studio interface for mapping data flows, create data flows from a pipeline or standalone, add sources and transformations, manage branches, and configure derived columns and filters.

Explore the data flow debug feature in mapping data flows, which spins up a serverless spark cluster via the Auto Resolve integration runtime and enables data previews and code view.

Implement a mapping data flow in Azure Data Factory Studio by creating a pipeline, adding a data flow with a CSV source, configuring data sets, and enabling data flow debug.

Publish and run a mapping data flow pipeline to copy and transform Celeste sales data, using derived columns and aggregates to produce a single file per store and month.

Explore schema inference and schema binding in mapping data flows within Azure Data Factory. Compare early versus late binding and address schema drift at runtime.

Discover how to create an Azure Data Factory data flow that transfers data from Azure SQL DB to a comma-delimited file, with early binding, late binding, and schema drift.

Learn to monitor data flow performance by optimizing cluster startup, source read, transformation, and sink write times, using parquet formats, partitioning, and broadcasting where appropriate.

Explore mapping data flows in Azure Data Factory, using built-in transformations such as filter, aggregate, and derived column to author pipelines, preview data, and manage schema binding, drift, and performance.

Explore data flows to enable reusability in pipelines, learn the flow UI, build a flow let to address a common data pattern, then debug and integrate it into a pipeline.

Explore flowlets in Azure Data Factory, reusable containers for modular data transformations with inputs and outputs, including mapping data flows, and learn to debug and apply common patterns.

Explore data transformation patterns for flow bullets and flow nets, including trimming, formatting, deduplication, and validating data before warehousing.

Navigate Data Factory Studio to build a flow by converting selected transformations, and add flow lits from the data flow section in factory resources.

Build a demo flow lit with an input, a derived column, and an output to format postal codes as six-character strings padded with zeros.

Convert the filter transformation into a flowlet to remove zero sales quantities, configure inputs (remove transaction id/date, set quantity to long, add filename) and test with debug parameters before publishing.

Explore data quality management in a pipeline by reviewing asset transformation, completing a hands-on exercise to improve data quality, and implementing error handling to control pipeline flow.

Explore asset transformation in Azure Data Factory to enforce data quality and validation in mapping data flows, using asset types: expected true, expect unique, and expect exists, to flag errors.

Implement an asset transformation to validate that the sales region is EU or UK using an assert transformation with Expect true, flagging nonconforming rows as errors in the data flow.

Identify error rows using an assert transformation and a derived column to create is error row and has error row fields for sales region failures (not UK or EU).

Apply an assert transformation to identify error rows where the sales region is not EU or UK, split clean and error streams, and save errors to an error log file.

Explore error handling in Azure Data Factory to control pipeline flow with on success, on failure, on completion, and on skip states. Implement try-catch and do-if-else patterns for graceful exits.

Implement error handling in Azure Data Factory by adding a fail activity with an on-fail dependency to capture and log errors to a file.

Capture pipeline failures by storing the error message and error code in two pipeline variables via set variable activities triggered by on failed dependencies from the fail activity.

Learn to log pipeline errors in Azure Data Factory by writing error message and error code columns to an error log file, using a copy data activity.

Demonstrates implementing error handling in Azure Data Factory by simulating a missing source file, using a fail activity to capture error message and code, and writing to an error log.

Transform all Celeste files with a data factory pipeline by cloning and updating data flows and datasets, copying from raw to cleansed containers, and fixing file naming and path handling.

Build efficient data pipelines by using pre-built templates to move files from the root to the sales data folder in Azure Data Factory, with filtering and deletion steps.

Explore asset transformation benefits and control pipeline flow with conditional logic. Build robust data quality and error handling in a data pipeline, and learn data warehouse design with Data Factory.

Explore Azure Data Factory basics, design a data warehouse for Veneer World, compare two data warehouse approaches, examine the data model and processing methods, and build the staging layer.

Discover data warehouses as a single source of truth for multi-source historical and analytical reporting, with integrated data, standardized quality, and denormalized tables for fast analytics.

Explore two common data warehouse models, Kimball and Inman, and compare centralized enterprise data warehouse designs with data marts, conformed dimensions, and data vault modeling.

Explore the Kimball-based data warehouse using a star schema with a monthly-grain fact table and conformed dimensions—date, store, territory, currency, and product, with surrogate keys.

Compare ETL and ELT data processing for data warehouses, then implement ELT by extracting to the Azure data lake, cleansing, staging, and transforming into dimensions and facts.

Create the stage and data warehouse tables in Azure SQL DB using git-sourced scripts, loading cleansed ADLS Gen2 data into the stage schema and executing in Azure Data Studio.

Build the data warehouse by creating dimension and fact tables using Azure Data Studio and SQL scripts, verifying empty tables, and preparing for data loading in the next lesson.

Build an Azure Data Factory pipeline that copies five master data files from the raw to the cleansed container using a delimited text dataset, with wildcard path and parallel copy.

Copy product data from raw to cleansed containers with Azure Data Factory pipeline using ADLS Gen2 CSV. The next lesson loads these files into staging tables in Azure SQL DB.

Use a metadata-driven approach to load cleansed files from the Azure Data Lake into staging tables in the Azure SQL DB, using a single source and sink data set.

Create parameter driven datasets in Azure Data Factory to load csv files from cleansed container into staging tables in Azure SQL DB, using metadata for folder, file, delimiter, table name.

Build a metadata-driven pipeline to load master data and product data from the cleansed container into Azure SQL staging tables, using parameterized datasets and dynamic mapping.

Execute a metadata-driven pipeline to populate Azure SQL DB staging tables from master data and product data stored in ADLS Gen2, validating load results and cost metrics.

Build a metadata-driven pipeline to load sales files into three store staging tables in Azure SQL DB, manage staging clearance with a lookup, and prepare for a single stage table.

Combine product data from staging tables into a single staging product table by building and publishing an Azure Data Factory pipeline that runs a stored procedure in Azure SQL DB.

Combine three store staging sales tables into a single staging table using the USB load stage sales procedure in Azure SQL DB, then publish and run Azure Data Factory pipeline.

Explore data warehouse approaches and the star schema model, learn the ELT method, and stage master and transactional data with a generic Azure Data Factory pipeline and Azure SQL objects.