Master Data Engineering using Azure Data Analytics

This lecture demonstrates how to install and launch VS Code on Windows 11, with guidance for Windows 10, and set up VS Code as the course IDE.

Install Python 3.9 on Windows using the 64-bit installer from python.org, verify the install with a hello world, and preview launching the Python CLI with PowerShell.

Set up the Python 3.9 path on Windows by updating the system path and verifying in PowerShell, then launch the Python CLI and print Hello World.

Set up Python 3.9 on Windows and install the VS Code Python extension to integrate Python with VS Code, create a workspace, and run a hello world program.

Sign up for the Azure portal using a browser, create a portal account, verify your email, and claim a $100 credit for one month.

Sign up for an Azure subscription, claim the $200 credit via a free trial, complete verification and payment steps, and review billing and cost management in the portal.

Learn to use Azure CLI and Azure Cloud Shell to manage resources, including launching Cloud Shell, selecting a subscription, and listing storage accounts via Bash or PowerShell.

Install the Azure CLI on Windows, Mac, or Ubuntu using MSI or package managers, validate the installation with PowerShell, and learn core commands to integrate with your Azure portal account.

Configure the Azure CLI against a specific Azure portal account using az login and logout, then verify storage accounts and subscription via Cloud Shell.

Explore cost management and billing in the Azure portal by reviewing invoices, updating payment methods, and analyzing costs with budgets and cost alerts.

Launch Azure Cloud Shell and review the initial storage account and resource group it creates, then understand how storage accounts relate to resource groups and containers.

Create an Azure resource group in the portal by selecting a subscription, naming it rg retail, and choosing a region, then review and create.

Learn how to add a storage account to an Azure resource group by using the marketplace wizard, selecting subscription, resource group, and standard locally redundant storage, then review and deploy.

Learn how Azure resource groups organize subscriptions and resources, including storage accounts and cloud shell, and how to group data workflows like ADF and data warehouse within a resource group.

Download the retail_db dataset from the GitHub repository, extract the zip to a local data folder, then upload the prepared data to your storage account container for data sharing.

Learn how to create a data container in an Azure storage account and prepare for uploading datasets from your local file system, emphasizing upfront container creation.

Upload the retail db folder to an Azure storage account via the Azure portal, highlight the single-file limit, and adopt Storage Explorer for multi-file or folder uploads.

Set up Azure storage explorer on Windows or Mac, sign in to Azure, and learn to manage files and folders in storage accounts and containers, including uploading local files.

Learn to access storage containers with Azure Storage Explorer, upload a local folder into a data container, and verify success by opening files in a code editor.

Validate that files uploaded via Storage Explorer appear in the Azure portal by accessing the storage account and data container, and note that Storage Explorer is easier for file management.

Create an adls storage account in Azure by selecting the adls Gen2 configuration, choosing a resource group, and reviewing details via the resource page, storage browser, and containers.

Upgrade blob storage to Azure Data Lake Storage Gen2 to enable external tables in Synapse and coexist with Databricks. Validate, disable default retention settings, and complete the upgrade.

Set up azure data factory and launch edf studio, create linked services and datasets, build a simple data flow from csv, and trigger a pipeline for etl.

Set up a new data factory in the chosen subscription and resource group, ensuring a unique name. Launch the ADF Studio to develop and deploy pipelines and data flows.

Explore Azure Data Factory Studio essentials, including data flows, pipelines, datasets, and Power Query, and learn to monitor, manage linked services and integration runtime, and build end-to-end pipelines.

Learn to create an ADF linked service to a blob storage data store, create datasets, and build pipelines while validating the connection within the same subscription and resource group.

Create your first ADF ETL pipeline by configuring a linkage service to blob storage, creating a CSV dataset, and previewing data before publishing.

Review key properties for CSV datasets in ADF, including column delimiter, row delimiter, encoding, header handling, and null value representations, then preview, validate, and publish to data flow and pipeline.

Create a parquet dataset in blob storage for orders park. Connect to the ITV retail storage account, specify the file path, test the connection, and publish for the data flow.

Learn to define the data schema from csv and sql sources by identifying column names and data types for datasets like orders and auditors, then implement in adf data flow.

Create a data flow with an order CSV dataset, add a CSV source, and define the schema. Preview the data with debugging to validate column names and types before publishing.

Define a cache sink in the data flow to validate reading csv data and writing to a file format, then publish and validate to catch errors.

Create the pipeline, rename it as file format converter pipeline, and rename the data flow to file format converter orders. Publish changes, adjust settings, run the pipeline, and validate results.

Trigger and monitor an EDF dataflow pipeline in a serverless environment, reviewing runs and lineage as infrastructure provisions. View pipeline details, switch to parquet, and learn compute size and performance.

Update the ADF data flow to sink into an ADLS parquet dataset, persist data to storage, publish, and run the pipeline to verify the parquet files and schema.

Start Azure Data Factory by creating a linked service to blob storage, configuring datasets for source (CSV) and target (Spark), and building a data flow plus pipeline to run ETL.

Execute an exercise to build an Azure data factory workflow for order items, creating source and target datasets, a data flow, and a pipeline to convert CSV to parquet.

Learn how to implement ETL logic with ADF data flow to compute daily revenue, join orders and order items, filter complete or closed orders, and trigger pipelines.

Create a data flow to compute daily product revenue with Azure Data Factory, adding the orders source, configuring projection, previewing data, and publishing while applying joins and aggregations.

Learn how to filter data in Azure Data Factory data flow to keep only complete or closed orders, join with order items, and perform aggregation to compute daily revenue.

Create an ADF pipeline to validate the data flow and compute daily revenue, trigger and publish the pipeline, then review the run and compare default versus custom integration runtime.

Create a custom Azure Data Factory integration runtime to run ADF pipelines, configure dataflow runtime, set time to live, and validate pipelines efficiently without idle startup delays.

Validate a custom ADF integration runtime by triggering a pipeline to run a data flow, publish changes, and compare startup times against the default runtime.

use the in function in a data flow filter with the expression builder to compare order_status to complete or closed, then publish and run the pipeline on a custom runtime.

Explore setting up an ADF data flow join between orders and order items, including schema definition, inner join with order status filter, and sanity checks in a pipeline.

Run the ADF dataflow pipeline to validate the join between filtered orders and order items, review execution counts, and perform sanity checks before computing revenue in the next lecture.

Leverage an ADF data flow aggregate transformation to compute daily product revenue by grouping on order date and product ID, then summing order item subtotal and rounding to two decimals.

Learn how to configure an ADF data flow sink to write daily product revenue results to Azure storage in Parquet format with snappy compression, including dataset creation and publishing changes.

Trigger the Azure Data Factory pipeline with ETL data flow to compute daily revenue from orders and audit times, save output to storage, and review the pipeline run details.

Access the JSON representation of ADF data flows and pipelines, view or copy the code, modify only via graphical interface, and share via git to collaborate efficiently.

Enable dynamic execution of Azure Data Factory pipelines by parameterizing the data flow to compute daily revenue and support T minus one or T minus two days.

Create an Azure Data Factory dataset that uses a parameter to dynamically generate monthly folders in the target location, leveraging the order_month parameter and parker file format.

define and apply parameters in an adf data flow to drive filter transformations, including setting a default order month and using runtime parameters in filters to segment data by month.

Create an Azure Data Factory pipeline with parameters, propagate them to the dataflow and dataset, enabling dynamic runtime execution and triggering of the pipeline.

Run an ADF pipeline with dynamic parameters, validate results for 2013 July and 2013 August, and review generated files and dataset previews to ensure correct filtering and data.

Explore common Azure data factory pipeline activities, including general, iteration, and conditionals; learn to trigger pipelines, execute procedures, and use for each, if, switch, and until for robust workflows.

Explore how to configure an Azure Data Factory ForEach activity by parameterizing an array, passing items with dynamic content, and using current item in a wait sequence to run iterations.

Execute a for each driven baseline ETL load in Azure Data Factory, parameterizing by order_month and passing each month to a child pipeline that computes daily product revenue monthly.

Trigger the ADF pipeline for the baseline load, pass multiple months, and validate results by reviewing run details, storage folders, and dataset previews for 2013 September and October.

Explore performance tuning of ADF data flows and pipelines, focusing on integration runtime, bottlenecks, and validating improvements to baseline ETL loads.

Choose the right integration runtime and compute size for data flows based on workload and time to live, using default or custom runtimes with general purpose or memory optimized compute.

Identify performance bottlenecks in a baseline ADF pipeline by analyzing pipeline runs, cluster startup time, and dataflow runtime, and optimize with a custom integration runtime.

Speed up your data pipelines by using a custom integration runtime with a non-zero time to live, reducing cluster startup time and speeding up the baseline load.

Learn to run an ADF pipeline in parallel with a ForEach activity and tune batch counts. Switch from the default to a custom integration runtime to reduce baseline time.

Troubleshoot shuffling and too many small files in daily revenue pipelines by tuning partitions and set partitioning to reduce file counts and improve performance in Azure Data Analytics.

Reduce shuffle partitions in the ADF data flow aggregate by replacing the default 200 patterns with smaller values using set partitioning and round robin to boost performance and decrease files.

Tune the performance of EDF data flows and pipelines, reducing the baseline load from one hour to one minute via a custom integration runtime and optimized shuffling.