DP-203 : Microsoft Azure Data Engineer Ultimate Course 2025

Today we will discuss what is Azure Data Factory, it's use cases and also, we will try out copying data from one storage account to another using Azure Data Factory ( ADF ) .

In this hands-on lecture, we will guide you through the process of creating an end-to-end Azure Synapse Analytics workload project from scratch. This comprehensive tutorial is designed to demonstrate big data workloads by converting data frames to CSV and Parquet files.

What You'll Learn:

• Creating a Synapse Analytics Workspace: Step-by-step instructions to set up your workspace with integrated Data Lake Storage Gen2.

• Creating an Apache Spark Pool: Learn how to create and configure an Apache Spark pool using the Azure Portal for high-performance big data processing.

• Creating a Notebook: Detailed guidance on creating and managing notebooks in Azure Synapse Studio to interact with Apache Spark.

• Running Spark SQL Queries: Execute Spark SQL queries, list tables, display data, and visualize results using charts.

This lecture provides in-depth explanations and practical examples to help you master Azure Synapse Analytics and effectively manage big data workloads.

In this comprehensive hands-on lecture, we demonstrate how to ingest semi-structured data, specifically JSON files, into an Azure SQL Database using Azure Data Factory. This end-to-end tutorial is implemented and demonstrated using an architecture diagram for clarity.

What You'll Learn:

• Ingesting JSON Files: Step-by-step guide to ingesting .json input files from blob storage.

• Configuring Data Factory: Detailed instructions on setting up data sets and pipelines in Azure Data Factory.

• Network Configuration: Learn how to configure network parameters for the entire architecture.

• Setting Up Azure SQL Database & Server: From scratch setup of Azure SQL Database and Server.

• Data Validation: Validate the data using both Azure Data Factory and Azure SQL Database to ensure there is no data loss.

In this hands-on lecture, we guide you through mapping data flows using Azure Data Factory. You'll learn how to create a storage account to serve as the landing location for input in .csv files and the staging area for output files.

What You'll Learn:

• Creating a Storage Account: Step-by-step instructions to set up a storage account for input and output files.

• Data Flow Debug: Use Azure Data Factory to infer the DDL and schema of the .csv input file.

• Creating Data Flows: Build data flows within Azure Data Factory to perform ETL operations such as select, filter, and sort.

• Using Data Factory Pipeline: Trigger the data flow and obtain the output file in .csv format in the staging area.

This lecture provides in-depth explanations and practical examples to help you master data flow mapping and ETL operations with Azure Data Factory.

In this hands-on tutorial, we guide you through the process of creating an Event Hub from scratch using an ARM template. This comprehensive video covers every step, from preparing the ARM template on your local computer to deploying it via Azure Cloud Shell.

What You'll Learn:

• Preparing the ARM Template: Understand the structure and components of the ARM template in a .json file.

• Uploading to Cloud Shell: Step-by-step instructions to upload the ARM template to Azure Cloud Shell.

• Executing Deployment Commands: Learn the exact Cloud Shell commands to deploy the Event Hub.

• Verifying Deployment: Check and validate the successful creation of the Event Hub in the Azure portal.

In this comprehensive hands-on tutorial, we guide you through the process of implementing dynamic data masking in Azure SQL Database to protect personally identifiable information (PII). Learn how to ensure that only authorized users can view PII data, while others see masked data even when running the same queries.

What You'll Learn:

• Creating Azure SQL Database and SQL Server: Step-by-step instructions to set up your database and server.

• Configuring Firewall Rules: Learn how to configure firewall rules to secure your database.

• Creating Users in the Database: Detailed guidance on creating different users and managing permissions.

• Granting Permissions: Understand how to grant appropriate permissions to users.

• Creating Tables with Masked Columns: Learn to create tables with masked columns to protect PII data.

• Inserting Values into the Table: Step-by-step process to insert data into the masked columns.

• Demonstrating Dynamic Data Masking: Watch as we demonstrate querying the same data as different users to show how dynamic data masking works.

Join us to gain practical experience and master the implementation of dynamic data masking in Azure SQL Database, ensuring data security and compliance with privacy regulations.

In this comprehensive hands-on tutorial, we guide you through the process of creating and configuring an Azure Log Analytics Workspace from scratch. This video covers every step, from initial setup to advanced configurations, ensuring you gain practical experience with Azure's powerful logging and analytics tools.

What You'll Learn:

• Creating Azure Log Analytics Workspace: Step-by-step instructions to set up your Log Analytics Workspace.

• Configuring Diagnostic Settings: Learn how to configure diagnostic settings to set up the destination for logs.

• Querying Log Analytics Data: Detailed guidance on querying log analytics data to extract valuable insights.

• Building Charts and Dashboards: Create and configure charts and dashboards to visualize data across multiple Log Analytics Workspaces.

Join us to master the creation and configuration of Azure Log Analytics Workspaces, and enhance your skills in data logging, analysis, and visualization with real-world examples.

In this comprehensive hands-on tutorial, we guide you through the process of analyzing IoT data using Azure Stream Analytics from scratch. This video covers every step, ensuring you gain practical experience with Azure's powerful analytics tools.

What You'll Learn:

• Creating a Stream Analytics Job: Step-by-step instructions to set up your Stream Analytics Job.

• Configuring the Job to Query IoT Data: Learn how to configure the job to query IoT data

• Uploading Sample Input: Upload a sample IoT data file in JSON format from your local computer.

• Executing and Testing Queries: Run and test the queries to analyze the IoT data effectively.

Join us to master the process of analyzing IoT data with Azure Stream Analytics, and enhance your data analytics skills with real-world examples.

Dive into the world of Azure Data Explorer with our comprehensive hands-on tutorial. This video takes you through every step, from setting up Azure Data Explorer in the Azure portal to mastering KQL syntax and visualizing your data. Perfect for beginners and those looking to deepen their knowledge, this tutorial covers:

Creating Azure Data Explorer: Step-by-step guide to setting up Azure Data Explorer in the Azure portal.

Database Creation: Learn how to create a database for Azure Data Explorer.

Data Ingestion: Demonstration of ingesting sample Excel data from your local computer into Azure Data Explorer. KQL Basics: Introduction to KQL syntax to help you query and analyze your data effectively.

Exploring Result Grid and Visualization: Navigate the result grid and create visualizations in Azure Data Explorer.

Exporting Data: Detailed instructions on exporting a subset of data in a .csv file from Azure Data Explorer to your local computer.

Welcome to basics hands-on tutorial for Azure Analysis Services. In this video, we cover everything you need to know to get started and make the most out of Azure Analysis Services.

Here's what you'll learn:

1. Basic Theory Concepts: Understand the foundational concepts of Azure Analysis Services.

2. Creating Azure Analysis Services Server: Step-by-step guide to creating an Azure Analysis Services server using the Azure portal.

3. Adding a New Model: Learn how to add a new model after the deployment is complete.

4. Scaling and Pricing Tiers: Explanation of different pricing tiers and how to scale your Azure Analysis Services instance.

5. Replicas: Understand how replicas work and how to configure them for your needs.

Welcome to basics hands-on tutorial for Azure Analysis Services ! In this video, we cover everything you need to know to get started and make the most out of Azure Analysis Services using ARM template ( .json file ).

Here's what you'll learn:

1. Basic Theory Concepts: Understand the foundational concepts of Azure Analysis Services.

2. Creating Azure Analysis Services Server: Step-by-step guide to creating an Azure Analysis Services server using ARM Template .json file.

3. Scaling and Pricing Tiers: Explanation of different pricing tiers and how to scale your Azure Analysis Services instance.

4. Replicas: Understand how replicas work and how to configure them for your needs.

Join us for this comprehensive tutorial and become proficient in using Azure Analysis Services for your data analysis and business intelligence needs.

Welcome to our in-depth hands-on tutorial for Azure Data Share. In this video, we cover everything you need to know to get started and effectively use Azure Data Share.

Here's what you'll learn:

1. Theory Concepts of Azure Data Share: Understand the foundational concepts and benefits of using Azure Data Share.

2. Creating Azure Data Share Using Azure Portal: Step-by-step guide to creating an Azure Data Share using the Azure portal.

3. Inviting Data Scientists via Email: Learn how to invite data scientists to share analysis files using Azure Data Share.

4. Creating a New Storage Account and Container: Discover how to create a new storage account and container to store the analysis files that need to be shared with data scientists.

5. Email Invitation Process: Detailed explanation of the email invitation received by data scientists and step-by-step instructions on what they need to do to view the shared files.

Welcome to our in-depth hands-on tutorial for Azure Data Share using ARM Template ( JSON File ) . In this video, we cover everything you need to know to get started and effectively use Azure Data Share using ARM Template.

Here's what you'll learn:

1. Theory Concepts of Azure Data Share: Understand the foundational concepts and benefits of using Azure Data Share.

2. Creating Azure Data Share Using Azure Cloud Shell Commands: Step-by-step guide to creating an Azure Data Share using the Azure Cloud Shell Commands.

3. Inviting Data Scientists via Email: Learn how to invite data scientists to share analysis files using Azure Data Share.

4. Creating a New Storage Account and Container: Discover how to create a new storage account and container to store the analysis files that need to be shared with data scientists.