Microsoft DP-203 Certified: Azure Data Engineer Associate

Explore how data engineering builds systems for collecting, storing, and processing big data across structured, semi-structured, and unstructured formats to enable visualization and insights.

Explore the four storage types inside an Azure storage account: blob for binary files, table services for structured NoSQL data, file services for shared files, and queuing services for messaging.

Log in to the Azure portal, create a storage account by selecting subscription, resource group, region, and performance and redundancy options, then review and deploy.

Discover how to create an Azure Data Lake Gen2 storage account with hierarchical namespace enabled, compare it to a simple storage account, and enable access control lists for security.

Explore Azure SQL pricing tiers and configurations, including DTU-based basics, standard, premium, and vcore-based general purpose, serverless, hyperscale, and business critical options; plus Azure hybrid benefits for discounts.

Learn to create Azure SQL database by logging into the portal, selecting a resource group and server, configuring admin credentials, choosing pricing tiers, and using elastic pools to optimize costs.

Learn to use Azure Data Studio to query AdventureWorks product table with schema, select columns, alias name as product name, save results as csv, json, or markdown, and visualize chart.

Explore the T-SQL partition by statement using sales order details and product IDs to produce per-row totals, enabling drill-down analysis and percentage calculations without collapsing to a single group.

Master how to use common table expressions with the with clause in T-SQL to name a query and reuse its result across subsequent SQL statements.

Explore Cosmos DB APIs, including the MongoDB API and six options—SQL API, Mongo database API, Mongo database, table API, Cassandra API, and Gremlin API—and how they support flexible data modeling.

Explore data modeling styles and Cosmos DB APIs, including document model with SQL and MongoDB APIs, JSON key-value form, table API, Cassandra columnar API, and Gremlin graph API.

Explore how Cosmos DB automatically partitions data into physical partitions within a container, distributing items across partitions with up to 10,000 request units per second and 50 GB per partition.

Learn how to set and manage Cosmos DB consistency levels, including eventual, consistent prefix, session, strong, and bounded staleness, by adjusting settings and specifying maximum lag time for read-write synchronization.

Master Cosmos DB basics, pricing, throughput units, and Mongo, table, and Cassandra APIs; learn containers, items, partitions, consistency levels, and global scaling through hands-on labs.

Explore Azure Synapse Analytics, an OLAP platform unifying data ingestion, storage, and analytics from sources, with query across data lake gen2, sql data warehouse, and spark pools for machine learning.

Understand Synapse SQL architecture with dedicated and serverless pools, a control node using MPP to allocate 60 compute nodes, with storage scaled separately and distributions like hash, round robin, replicated.

Explore the overview of dedicated SQL pools, compare with serverless options, and learn how dedicated compute and columnar storage enable external and normal tables for data warehousing.

Learn how to move data from data lake storage into a dedicated sql pool using external tables for csv format, including hadoop vs native types and sas token authentication.

Identify the roles of fact and dimension tables in data warehousing, where fact tables hold quantitative data like sales amount and units sold, and dimension tables provide context.

Learn to create table distributions in a SQL pool by switching from round robin to a hash distribution on a chosen column, and note replicated distribution for a cap table.

Learn to monitor query performance in the dedicated SQL pool, identify slow operations, and optimize by using hash distribution on group-by columns to speed up joins and aggregates.

Understand how a business key and surrogate key support slowly changing dimensions in dimension tables, using identity columns, non-null constraints, and proper mapping during data integration.

Boost query performance using indexes, including cluster columnstore indexes, distribution types, and where clause optimization; choose heap for temporary data and plan when to use clustered or non-clustered indexes.

Explore Azure Synapse database templates by industry to quickly generate a data warehouse. Learn how to select automotive templates, add inventory tables, and configure primary and foreign keys.

Explore spark's master-slave architecture with a driver and executors across a cluster. Understand how spark context coordinates execution using resilient distributed datasets and a directed acyclic graph.

Transfer data from a csv file in the data lake storage to a dedicated sql pool in Synapse using spark and a sql connector, defining a schema, staging, and credentials.

Explore spark history and in-memory processing within Synapse Analytics, using spark pools and languages Python, spark SQL, R, and Scala, with labs on reading and writing data to SQL pools.

Explore the overview of Azure Data Factory, a cloud-based ETL tool for data integration. Learn how pipelines, activities, linked services, and integration runtime enable code-free data flows and monitoring.

Master the ETL process: extract data from sources like data lakes or SQL databases, transform it, and load into Azure Data Lake Gen2 or Azure Synapse for analytics.

Configure and deploy an Azure Data Factory in the portal by selecting a resource group, location, and networking options, then create and launch the Data Factory Studio to manage pipelines.

Learn to process JSON data with arrays and objects in Azure Data Factory data flow, using flatten and mapping to load into Synapse SQL from a data lake.

Select the appropriate integration runtime for Azure Data Factory and Synapse Pipelines—Azure, self-hosted, or Azure SSIS—based on data location and the need for data flow, data movement, or dispatch.

Install and configure a self-hosted integration runtime for Azure Data Factory, including downloading the installer, completing setup, registering with authentication keys, and understanding its data flows limitations.

Copy data from your pc to a data lake with a self-hosted integration runtime and a file system source, then publish and verify the json file.

Discover how azure data factory components are stored in a git repository as json files, with pipelines, activities, data set, and linked services represented in code.

Explore Databricks, a cloud service by Apache Spark creators for big data analytics and data engineering, featuring the data lake house for scalable governance on Azure and AWS.

Explore the benefits of Databricks, including ACID-compliant transactional support with atomicity, consistency, isolation, and durability, plus Delta Lake’s unified storage for diverse data formats and cost savings.

Create an Azure Databricks workspace and configure interactive and job-based clusters to run notebooks, then explore the Databricks file system and pinned compute to prevent 30-day deletion.

Explore compute cluster types in Azure Databricks, including single user and shared clusters, with driver and executor architectures, supported languages (Python, Scala, R), and cost considerations.

Create a compute in the databricks workspace to run notebooks and workflows, selecting all purpose compute, single node, spark 3.5.0, and appropriate storage, then monitor inactivity termination.

Learn to create a test database and table in a data lake, copy data with sql, merge the schema, and handle csv and parquet formats with header options.

Demonstrates streaming csv data from a data lake with Databricks, using spark.readStream and cloud files to process new files, validate schema, and update checkpoints.

Write streaming data from Databricks to Azure Synapse dedicated SQL pool, using a spark stream write to populate the DBO.Amazon table with a checkpoint for reliable processing.

Read data from a synapse dedicated sql pool using spark.read.format with a select from dbo.amazon, configuring workspace url, sql pool, credentials, staging, and master key for encryption.

Learn how delta tables version history works in a data lake, with parquet-based storage and automatic management, including querying six versions and rolling back to a chosen version.

Create an event hub inside a namespace, set a six-hour retention, and create a partition to host events in this hands-on Azure data engineer training video.

Delete unused resources to save costs by removing databases, Databricks resource groups, and virtual machines used for clusters, as demonstrated in this hands-on cleanup.

Create an Event Hub inside an Event Hub namespace to ingest events, set a name and partition count, apply a 24-hour retention, then review and create the hub.

Ingest data in real time into an Event Hub using Data Explorer, sending custom or demo payloads like Yellow Taxi, and monitor event counts and partitions for real-world applications.

Define inputs for a stream analytics job by configuring an event hub input, selecting json serialization, testing the connection, and preparing to route data to outputs.

Formulate a query using cross apply and get array value to extract nested fields like category and identity account name from JSON and Avro data in stream analytics.