[8 Course BUNDLE]: DP-203: Data Engineering on MS Azure

Learn to create an Azure SQL Database in the portal by provisioning a server within a resource group, configuring compute and storage, networking, backups, and deployment progress.

Compare dtu-based and vcore-based pricing tiers for azure sql database, selecting basic, standard, premium, general purpose, hyperscale, or business critical configurations, with provisioned or serverless compute and hybrid benefit.

Discover how failover groups enable automatic failover for disaster recovery and business continuity, with primary and secondary servers, read-write and read-only endpoints, and seamless data replication.

Enable auditing to track database activities and store audit logs in a storage account or log analytics. Configure retention days and review events like create table and select.

Learn how Azure Data Factory enables extract, transform, and load workflows. It provides cloud-based data integration through pipelines, provisioning, publishing, and monitoring.

Explore the building blocks of data factory—pipelines, activities, linked services, datasets, data flows, and triggers—and learn how to create and connect them in the portal.

Learn to use the get metadata activity in a data factory pipeline, creating a dataset and link service to scan a local folder and return file metadata.

Use the filter activity in a data factory workflow to filter file names from the get metadata output, selecting only those starting with E in a folder input.

Learn how to use the if condition activity in data factory to branch between true and false activities. Use an expression and the empty function to evaluate the previous output.

Define pipeline-level variables and use append variable to add values in an if condition, then access them across the pipeline and set defaults.

Iterate over the filter output using a for each loop to copy each file into a data lake, configuring inputs, sequential processing, and per-file copy activity.

Explore the dataflow source transformation in Azure Data Factory, configuring datasets from blob storage or data lake, and handling schema drift for scalable visual data integration.

Configure and apply the sink transformation to load data into a destination dataset, creating and mapping a school database table, then publish, run the pipeline, and verify records.

Explore adding a derived column in a dataflow to create a new field, such as a hardcoded customer name, and see auto mapping populate the sink.

Enrich sales data with a lookup transformation in a data factory data flow by joining on customer id to fetch names from a reference table; publish the pipeline.

Configure a filter transformation in the data flow to select records with quantity greater than four, publish changes, and validate results across the destination tables.

Explore the exist transformation in a data flow to filter records by existence in a second source, configure exist or doesn't exist, and validate results by running the pipeline.

Define parameters inside the dataflow and use them in transformations and expressions, with values passed from the calling pipeline and a default if none provided.

Explore monitoring data factory performance using a zone monitor, diagnostic settings, and log analytics to view logs, metrics, health, and troubleshoot unexpected behavior.

Implement an Azure Data Factory v2 lab that builds a pipeline to load source SQL data (customer and lead) into staging with incremental and full loads using lookup and filters.

Explore slowly changing dimension type 1, where updates replace existing records to avoid history in the dimension, using staging, incremental loads, and a surrogate key.

Implement slowly changing dimension type 1 in Azure Data Factory v2 using data flow, staging and dimension tables, exist checks, and insert/update logic.

Learn to implement slowly changing dimension type 2 in ADFv2 with a data flow, using staging and dimension, lookups, and conditional splits to insert new records and update current flags.

Explore Azure Data Lake Storage Gen2, a cost-effective, scalable solution built on blob storage for unstructured data, supporting ELT workflows and Hadoop-compatible analytics.

Create an Azure data lake storage Gen2 in the portal, selecting subscription, resource group, location, and storage tier, with hierarchical namespace and replication options, plus hot or cool access tiers.

Grant access via storage account and granular file or folder permissions, assigning roles like owner or contributor. Review data transfer options, including offline data box, rest api, and easy copy.

Explore how to monitor Azure Data Lake Storage using classic and monitoring views, configure alert rules for latency, capacity, and ingress/egress, and review diagnostic and usage graphs for admin insights.

Discover how to load data into ADLS Gen2 using Azure Data Factory, by building a copy pipeline from local files, configuring sources, destinations, and file formats.

Explore the Synapse analytics architecture, from the control node to isolated compute and storage nodes, and learn how hash, round-robin, and replicated data distributions power parallel queries.

Explore PolyBase in Azure Synapse to query data stored in blob storage or Hadoop by creating external data sources, file formats, and external tables mapping to your data lake.

Learn how to load data into a data warehouse table using the copy into statement, including source specification, sas token authentication, and skipping header rows from Azure blob storage.

Load data from a local file system into Synapse Analytics tables using Azure Data Factory, staging the file in blob storage and loading with Polybius or bulk insert.

Create and manage restore points to back up a Synapse SQL pool, then restore or create a new pool from the restore point, including timestamps and scheduling options.

Explore dual backup across regions for disaster recovery and business continuity, automatically enabling daily backups and simple restore by creating a new skill pool.

Explore Cosmos DB, a globally distributed database you can deploy worldwide with a few clicks, offering multi API support including SQL and Cassandra, across regional Azure data centers.

Explore Cosmos DB pricing by storage and requests, compare provisioned throughput options—standard, auto scale, and serverless—and learn how manual vs automatic scaling affects cost and performance.

Create a Cosmos DB account in the Azure portal by selecting a subscription, resource group, location, and API, then configure provisioned throughput, redundancy, and consistency level.

Explore how Cosmos DB supports multiple APIs—document, key-value, columnar, and graph models—through Cassandra API, Gremlin API, MongoDB API, and Table API, enabling flexible data modeling and seamless migration.

Learn the database container items hierarchy in cosmos db, from a database account to databases, containers, and items, and how stored procedures, user defined functions, and triggers live inside containers.

Explore how to monitor a Cosmos DB database using metrics, per-database and per-container views, region filters, throughput, requests, storage, latency, and cost.

Build a C# app to connect to Cosmos DB and read items from a container. Configure the endpoint and primary key, then read and write data from the database.

Create an Azure Databricks workspace in the portal by selecting your subscription, resource group, and workspace name in south central us, then access notebooks and tables inside the workspace.

Explore end-to-end data flow from source files in a blob storage account to a destination SQL database, guiding manual data upload, container setup, and basic transformation steps.

Extract data from a data lake by mounting containers, transform into global tables, and load only new customers by joining with dim customer to avoid duplicates.

Download and install Power BI Desktop from Microsoft Store on Windows or Mac, then connect to sources, build a data model, and create bar, line, and pie charts with filters.

Learn to create and manage table relationships, set cardinality, and configure cross filter directions in your Power BI data model.

Create calculated columns in Power BI with DAX, such as concatenating first and last names to form full customer names and deriving columns like tax amount divided by a value.

Use the power query editor to perform data cleansing and detailed transformations. Replace nulls with meaningful values via the replace values tool to improve data quality.

Apply the group by feature in the transform section to aggregate by city, state, and country using count, sum, average, max, and min, with an optional duplicate of the table.

Set the background color and add a company logo to a Power BI desktop report, while adjusting page background properties and inserting the logo image.

Add slicers for product category and order date to filter the report and explore performance by category and year and month across regions.

Explore building a city-level sales report by a selected product using a pie chart to display total sales by city, with formatting and legend placement to highlight city contributions.

Build detailed, table-based reports by adding customer and sales order data—quantity, unit price, discounts, and ship method—and use conditional formatting to reveal city-level sales and product-level insights.

Add and customize card visuals to show current and previous month sales using a previous month measure with order date filters for quick snapshots.

Explore page-level and all-pages filters, set a default city filter like London, and use slicers to synchronize visuals across a multi-page report before publishing to the cloud for sharing.

Install and configure the personal gateway for the Power BI service on your local laptop to access the Excel file and the Aja Sequel Database, enabling dataset refresh.

Publish a Power BI service report and share it with colleagues via email or links, configure access and collaboration options, and note row-level security as an advanced topic.

Create a Power BI service dashboard by pinning visuals from existing reports. Share the dashboard with others and understand its read-only snapshot nature.