Build a Data Mesh using Azure-Hands on, real-world scenario

Apply domain driven design to define data domains for a modern home financing data mesh, enabling decentralized data management where each microservice manages its own data stores.

Apply consumer aligned domains by matching with end user use cases in a mortgage context. Implement federated data governance across origination and servicing to ensure data quality, security, and compliance.

Analyze case study datasets for origination and servicing domains, including loan details, customer demographics, fico scores, payments, flags like autopay and two-factor authentication, to build your data mesh.

Build a data mesh on Azure by implementing bronze, silver, and gold data layers, ETL/ELT pipelines, and domain teams for origination and servicing, with federated governance and Azure Synapse.

Set up an Azure account to begin building a data mesh in Azure and understand how billing works. Create a free Azure account with card verification and identification to proceed.

Discover the four core Azure service categories—computing, analytics, storage, and networking—and learn pricing options like pay-as-you-go, volume and commitment discounts, plus free services and support plans.

Create an Azure account from azure.microsoft.com by starting free, claim $200 credit, sign up with a Microsoft or GitHub account, and verify by phone.

Decentralize data ownership within a data mesh by distributing ownership across the origination and servicing domains, and using Microsoft Entra ID to create teams, users, and groups for federated governance.

See how Azure Active Directory is now Microsoft Entra ID in the Azure portal, with no change to features, users, or groups; we’ll still call it Azure AD.

Learn how to create Azure AD users and groups in the Azure portal for a data mesh, covering origination, servicing, federated governance, and cross-domain analytics.

Sign in to portal.azure.com, access azure active directory, and create a new user or invite an external user in the domain, setting a password and reviewing the user’s details.

Create security groups in azure active directory to manage access by teams, assigning owners and members for scalable permissions. Dynamic membership requires premium licenses, while this scenario uses assigned membership.

Create a product centric data infrastructure for a data mesh by provisioning Azure storage and a Synapse workspace per data domain (origination and servicing) and decentralizing data ownership.

Create domain-based Azure resource groups for origination and servicing to manage resources by lifecycle and region, store metadata, and reserve an admin resource group for administration with optional tags.

create a general purpose v2 storage account in the Azure portal, configure basic settings, networking, data protection, encryption, and tagging for a scalable data mesh.

Explore Azure Synapse Analytics, an integrated enterprise analytics service that unifies big data and data warehousing, enabling serverless or dedicated SQL pools, Spark, and Data Explorer for end-to-end analytics.

Learn to create an azure synapse workspace from portal.azure.com, selecting subscription, resource groups, region, storage, and security settings, then review and deploy a serverless sql pool.

Learn to implement federated governance and access policies in Azure to support decentralized data mesh, assign governance and team access policies, and apply role-based access control (rbac).

Define Azure RBAC by mapping roles, scope, permissions, and assignments to users or groups, using roles like reader, contributor, owner, or custom roles, across management groups, subscriptions, and resource groups.

Learn to assign roles in Azure across subscription, resource group, and resources using access control, understand owner, contributor, and reader roles, and apply storage data contributor as needed.

Explore Azure network security concepts such as virtual networks, firewall, express route, and security center, and learn to configure vnets, subnets, and private endpoints to protect storage and data domains.

Explore how Azure encrypts data at rest and in transit using Azure Disk encryption, Azure SQL Database Encryption, and Azure Storage Service encryption, with key management via Azure Key Vault.

Log in to Azure Synapse Studio from your workspace or browser to access a unified workspace with data hubs, notebooks, dashboards, data flows, and pipelines.

Explore the data hub in Azure Synapse Studio as a central catalog to discover and manage data assets, view metadata, and link external data sources.

Explore the development hub in azure synapse studio, creating SQL scripts, notebooks, data flows, and spark definitions, importing sample codes from the gallery to accelerate development.

Explore the Integrate Hub in Azure Synapse Studio to design pipelines, connect data sources with link connections, and copy data across Azure storage, blob storage, and Cosmos DB.

Monitor Hub provides a unified platform to monitor and manage data pipelines, triggers, and integration runtimes, across SQL, Spark, and data explorer pools.

Explore the manage hub in Synapse Studio to allocate resources, manage security and access, and configure linked services, credentials, libraries, and git-based source control.

Explore Azure Synapse serverless SQL pools, a pay-per-use, auto-scaling, fault-tolerant query engine for ad hoc querying and data exploration without provisioning resources.

Discover dedicated SQL pools in azure synapse: a scalable, distributed sql engine with control and compute nodes, internal storage, and polybase support for processing large structured data.

Explore Azure Spark pools and serverless Spark in Synapse, enabling in-memory, distributed processing with Spark context and Yarn, and enabling Spark SQL and ML libraries with data lake storages.

Explore data explorer pools as a scalable compute resource for interactive querying of structured and unstructured data, complementing SQL and spark pools in Azure for near real-time insights.

Explore data in azure synapse to inform data mesh design, extract features for data domains, and learn to query csv, parquet, and json files, using the open rosette function.

Upload the loan servicing dataset to a new container in the default Azure Synapse storage account, then explore it in Synapse Studio to learn refreshing and viewing the data.

Explore data in the data lake using Azure Synapse serverless SQL pool and the openrowset function to read csv, json, or parquet files with header and delimiter options.

Explore explicit data type casting in Synapse to optimize repeated queries, using stored procedure sp_describe_first_result_set, width clause, and overridden types like bigint and smallint to reduce cost.

Learn how to apply utf-8 collation to columns and databases in synapse, avoiding conversion errors by specifying column-level collations and creating a database-level solution.

Create an exploration database in service explorer db, set the database-level collation, and define an external data source for the storage account to query loan_servicing.csv without full paths.

Learn to query a subset of columns using the with clause, reference by header names or by position, and manage header rows to shape the dataset schema.

Develop the servicing domain in a data mesh by ingesting and transforming data with Azure Synapse, using copy data tool and data statement, loading silver to gold with spark pools.

Establish decentralized data ownership in our data mesh by forming domain oriented, product centric teams for origination and servicing, and enable access with Azure Active Directory and Azure RBAC.

Provision bronze, silver, and gold containers in the servicing data domain storage and create a linked service in Azure Synapse with account-key authentication after disabling blob soft delete.

Use the copy data tool in Azure Synapse to load source data into bronze via ELT, then advance to silver and gold layers, monitoring pipeline with auto resolve integration runtime.

The lecture demonstrates transforming bronze-layer data into the silver layer using CETAS and CTAS in Azure Synapse, creating external data sources and file formats, and exporting to parquet in silver.

Log in to the Azure portal, open the Synapse workspace, and create an Apache Spark pool with memory-optimized nodes, set min and max to three, and enable automatic pausing.

Transform and load data from the silver to the gold layer using Spark, reading a parquet file from silver, filtering active accounts, and writing the results back to gold.

Create data ingestion and transformations for the origination domain, copying data into bronze and applying curation to silver and gold. Build pipelines and data flows to load silver and gold.

Explore decentralized data mesh with domain oriented, product centric teams owning origination and servicing data domains, using Azure Active Directory and admin login on supervision plane for ingestion and transformations.

Build and configure an Azure Synapse pipeline to copy originations data from the source to the bronze layer in originations data domain data mesh, using a copy activity and datasets.

Create a dataflow in azure synapse to transform originations bronze data into silver, using a source dataset and a purchase filter, then publish and run via a pipeline.

Build a data flow from silver to gold, add a derived column PMI flag using kltv, and publish a pipeline to run flow and load data into silver and gold.

Build a data mesh in Azure Synapse by creating and running a data flow pipeline to load origination data into silver and gold layers, with validation, debugging, and triggers.

Learn to create external tables and views over gold and silver data in Azure Synapse, using an external data source and parquet format to enable secure, queryable presentation layer access.