Master Microsoft Fabric: A Complete End-to-End Project- CICD

Create Azure resources, including a storage account, a Synapse Analytics service, a data lake storage account, and a spark pool, to support migration tasks and prepare data for Synapse notebooks.

Create an Azure storage account to enable Azure data lake storage, enable hierarchical namespace, select a resource group and region, and complete the create wizard.

Explore the basics of Microsoft Fabric, its problem-solving benefits for data engineering, and the core components, plus how to enable and use Fabric for a strong course start.

Trace the evolution of data architectures from traditional data warehouses to lakehouses, noting the shift to data lakes, decoupled storage and compute, and delta lake integration with Microsoft Fabric.

Explore lakehouse architecture uniting data lake and data warehouse features for storing and analyzing structured, semi-structured, and unstructured data with delta lake and its transaction log enabling ACID properties.

Explore why Microsoft Fabric is needed as a single stop shop for end-to-end data projects, unifying data factory, data engineering, data warehouse, real-time analytics, Power BI, and One Lake.

Microsoft Fabric is an all-in-one analytics solution for enterprises, unifying data movement, data science, real-time analytics, business intelligence, data lake, engineering, and integration in a single end-to-end platform.

Enable and access Microsoft Fabric by logging into app.fabric.microsoft.com, then verify tenant settings in the admin portal and enable Fabric for the organization or groups.

Understand how microsoft fabric licensing is capacity based, with f2 to f64 skus, power bi premium equivalence, storage costs, and pause options, plus pay-as-you-go and one-year reservation pricing.

fabric is an all-in-one SaaS platform with a home page of experiences. each experience tailors tools by role, so BI users see Power BI items and data engineers see notebooks.

Learn the latest fabric ui update, where experiences become workloads and can be viewed in fabric or Power BI. Create a workspace, then click workloads and tiles to build items.

Understand Microsoft Fabric terminology, including experiences, workspaces, and items, with examples from data engineering and Power BI, and how the fabric home page groups these capabilities.

Explore how One Lake provides a single tenant-level storage in Microsoft Fabric, storing all workspace data as Delta Lake using delta parquet files and avoiding data duplication.

Discover how Microsoft Fabric uses one lake as the single source of truth, with a serverless compute and delta parquet data shared across Spark, SQL, Data Explorer, and Power BI.

Explore Microsoft Fabric Lakehouse, a data architecture platform for storing, managing, and analyzing structured and unstructured data in one location, and learn its components and how to ingest data.

Explain how Microsoft Fabric workspaces organize workloads within a fabric tenant by using capacities for high intensive workloads and low intensive workloads, and by creating team workspaces with needed compute.

Enable the fabric free trial for 60 days, then create a new workspace named Fabric Trail to host shareable items across data engineering and Power BI experiences.

Acquire fabric capacity from Azure by using the admin portal, select a capacity, link a subscription and resource group, and pause when unused to control costs.

Understand workspace security in Fabric by assigning roles like admin, contributor, and viewer to control who can create and access resources, manage access, and incur costs.

Create a lake house in Synapse workspace to store, manage, and analyze structured, unstructured, and semi-structured data in delta format. It includes a semantic model and a SQL analytics endpoint.

Explore the lake house in fabric’s lake house explorer, comparing tables and files, and learn how managed and unmanaged data, including delta format tables and uploads, support the semantic model.

Upload files to a lake house, create a delta table, and verify data storage in One Lake data hub, including the delta log and metadata.

Upload an entire directory to create a delta table from multiple csv files with the same schema, name with underscores, and load to a new table, and include subfolders.

Explore the lake house sql analytics endpoint, a read-only surface to run sql queries, view data, and create views or stored procedures, while dml on tables is not supported.

Access the sql analytics endpoint from ssms by copying the connection string, authenticating to the lake house database, and exploring tables, views, and system views in LH fabric.

Explore the visual query feature in the sql analytics endpoint to perform data transformations with a low-code interface, view generated sql, and save results as views.

Use the OneLake File Explorer to access fabric items in the lakehouse from your desktop with a OneDrive-like experience, and authenticate, upload, download, and sync files and tables.

Explore Data Factory in Microsoft Fabric, a data integration tool that ingests, prepares, and transforms data from sources, and build an end-to-end pipeline to a data lake with dataflows Gen2.

Explore the Fabric Data Factory UI, compare pipelines, data sets, and Power Query with Azure Data Factory, and learn how triggers, activities, and connections differ.

Learn five methods to load data into a lake house, including direct upload, Data Factory pipelines and Data Flows Gen2, Spark notebooks, and shortcuts.

Compare fabric data factory with Azure data factory for ingesting on premise sql server data, using on premise gateway, link service, and pipeline to feed one lake.

Explore gateway types in Microsoft Fabric, including the on prem data gateway and the VNet data gateway, enabling ingestion of on premise SQL data.

Install the on premise data gateway from the Microsoft Fabric portal, select standard mode, register the gateway with a recovery key, and confirm it shows online in settings.

Learn to create a connection to an on-prem SQL Server in Microsoft Fabric using the on-prem data gateway, with basic authentication and a gateway status of online.

Create a data pipeline in the fabric portal to ingest on-prem sql data into the lake house, using lookup and for-each to copy data with dynamic table names and append.

Demonstrate end-to-end data integration in fabric data factory by reading on-prem SQL data into one lake via a gateway, creating a pipeline, and comparing fabric with Azure Data Factory.

Explore Dataflow Gen2, a low-code UI tool to ingest and transform data from multiple sources with Power Query interface, writing to Azure SQL Database, Lake House, and Azure Data Explorer.

Create data flow Gen2 in fabric to read from Azure Data Lake, transform with a custom column mapping state codes to names, and publish to a lake house delta table.

compare data flow gen2 in Fabric with Azure Data Factory data flows, showing how to call a data flow from a pipeline, set sinks, and note backend differences.

Discover how one lake acts as the single unified data lake for your organization, storing data for multiple analytical engines. Learn to create, delete, and update shortcuts within one lake.

Assess the status of loading data into the lake house, tracing uploads, pipelines, and data flow gen2, then create and use shortcuts in the one lake and Kql database.

Create shortcuts at the lake house level to reference data across domains without copying it, with automatic updates and support for S3, Azure Data Lake, Dataverse, and Google Cloud Storage.

Learn prerequisites to create a shortcut in Fabric, including source data, write authentication, and destinations like lake house or SQL database.

Create a shortcut in the lakehouse files by linking an Azure Data Lake Gen2 source and a DFS path, keeping data synchronized without an ETL pipeline.

Create shortcuts in the tables section of Microsoft Fabric Lakehouse, noting that shortcuts must be top level and in delta format; subdirectory level data are not supported.

Upload the required parquet file to the project container and grant the synapse managed identity storage blob data contributor access, then verify storage permissions for notebook execution.

Create a delta format shortcut in the lake house table section by provisioning the shortcut delta route container and writing delta data via a spark notebook in Azure Synapse Analytics.

Create a delta file in Azure Synapse using a Spark notebook. Read a Parquet file from storage, then write it in Delta Lake format to a new container.

Create a shortcut in the tables section using a delta lake source from a top-level delta folder. Verify the shortcut table shows delta format data parsed from parquet.

Demonstrates that delta format data must reside in the top-level container for a shortcut table to be recognized, even when created under a subfolder, in Fabric.

Learn how to create a parquet format shortcut in a lake house table and why delta format is required at the top folder level for Delta Lake.

Learn the prerequisites for creating shortcuts in the tables and files sections of Microsoft Fabric, including top-level placement, delta parquet data, and when to use files versus tables.

Learn how update scenarios affect lakehouse to datalake shortcuts: with read access you cannot update, while with write access updates to the lakehouse propagate to the data lake via notebooks.

Demonstrates updating data in the data lake and syncing it to the lakehouse using Fabric shortcuts, with a Synapse notebook walkthrough showing automatic data refresh and latest counts.

Explore shortcut deletion scenarios in the lake house and data lake, examining outcomes when deleting files and tables, and when removing the shortcut entirely.

Demonstrate deletion in the fabric lake house by removing a shortcut file, and show how the data lake end reflects the change, including the mp1 csv content.

Demonstrates deleting specific content in ADLS, showing how removing rows or an entire MP two dot csv file affects the lake house data and its shortcuts.

Delete 112 rows from shortcut delta table in lakehouse using fabric notebooks, verify counts before and after (1524 to 1412), and confirm deletions reflect in Azure data lake Gen2.

Delete delta table data in adls and observe near real-time updates as 348 rows with education level high school diploma are removed, reducing 1412 to 1064 in the lake house.

Explore how deleting an entire shortcut affects data in Azure Data Lake by removing references rather than the underlying data in Delta Lake or source files.

Understand shortcut deletion scenarios in Microsoft Fabric, learning how deleting file shortcuts, targets, or data lake content depends on permissions and impacts data in the lake house.

Explore synapse data engineering within the lake house, learning how to collect, transform, and analyze data using spark pools, notebooks, and lake house tables.

Ingest data into the lakehouse using notebooks, configure spark, and explore the user interface and its elements from scratch to master data engineering workflows.

Discover how Microsoft Fabric automatically provisions a default spark pool for each workspace, enabling fast spark initialization and notebook execution for large-scale analytics in a lakehouse.

Explore spark pools in Microsoft Fabric, including the starter pool and custom pools; startup pools provide always-on spark clusters, with rapid initialization and billing only when active.

Explore spark pool node sizes in fabric from small to extra large, with 8 v cores and 64 gigabytes per medium node, plus starter pool defaults and dynamic allocation.

Customize the startup pool in workspace Spark settings by adjusting the node count or creating a custom pool, with auto scale and executors enabled.

Learn to create a custom pool in a Microsoft Fabric workspace by selecting node family and size, understand non pre-warmed sessions, and compare startup vs custom pools for testing.

contrast standard and high concurrency sessions in fabric notebooks, showing how high concurrency allows sharing a session across notebooks, speeds up initializations for subsequent notebooks, and improves single-user resource efficiency.

Change Spark settings to switch to the startup poll for rapid Spark session initialization, avoiding the custom poll.

Explore the front-end notebook UI in fabric portal, mastering save, download, autosave, run, schedule, and lake house integrations for PySpark workflows.

Explore fabric notebook basics: use the lake house, drag and drop tables and files, auto generate PySpark or Scala code with built-in code snippets and collaborative development with magic commands.

Explore MSSpark utils, an inbuilt Microsoft package offering common notebook tasks. Learn to work with file system, environment variable utilities, secrets, and credentials in Fabric and Synapse notebooks.

Explore the ms spark utils fs utilities, compare mount points with shortcuts, and learn to mount azure data lake storage with account key, then list mounts using a shortcut.

Learn how to use MSSparkUtils fs utilities to manage files in a lakehouse, including listing, creating folders, writing and viewing files, copying, moving, deleting with recursion, and existence checks.

Learn how MSSparkUtils fs fast copy accelerates large data transfers using Azcopy, comparing fast copy to normal copy and delivering dramatic time savings for 76 million records.

Create folders in microsoft fabric to organize notebooks and other items, then move them into a notebooks folder or subfolders to improve workspace clarity.

Explore ms spark utils notebook utilities to run and exit notebooks, pass parameters via parameter cells, and understand run versus exit behavior across sessions.

Execute multiple notebooks in parallel with Ms. spark utils notebook run multiple, including dependency-based sequencing and parameter passing, then view execution snapshots and optimize spark compute resources.

Learn to ingest azure data lake data into a lakehouse via notebooks, using a spark session, with authentication through Microsoft intra ID user or service principals with key vaults.

Authenticate notebook access to Azure Data Lake Gen2 with Entra ID by granting the storage blob data contributor role in the fabric portal. Read data and discuss production RBAC challenges.

Learn to authenticate to Azure data lake gen2 with a service principal, grant storage blob data contributor role, and manage app id, tenant id, and secret key in key vault.

Access adls using a service principal and azure key vault to securely retrieve app ID, tenant ID, and secret key, then write data to the lake house with fabric.

Call a fabric notebook from a fabric pipeline using a service principal to access Azure Data Lake via a notebook activity, with key input limits and planned key vault integration.

Compare managed and external tables in Microsoft Fabric, focusing on metadata and data storage, and learn creation methods: dot save as table, SQL create table, and Delta Table Builder API.

Create managed tables in a lakehouse using the catalog to identify managed versus external tables, and learn to build them via save as table, sql, or delta builder api.

Create an external table by storing data under a specified location using delta format or the location keyword. Distinguish external from managed tables by their files folder and delta log.

Discover when a shortcut table acts as a managed table instead of an external table, as the fabric engine references delta-formatted data stored externally.

Discover the data wrangler in fabric notebook, a GUI low-code tool that creates and analyzes data frames, sorts data, finds duplicate rows, and generates notebook code.

Explore environments in fabric by creating custom pools or startup configurations, and manage workspace-level spark settings to tailor runtimes, libraries, and workloads.

Learn how v order, a default reorder optimization in Microsoft Fabric, writes Parquet files with special sorting and row grouping to boost compression and read performance across Fabric engines.

Learn to define and run spark jobs offline using spark job definition, upload your Python file, reference a lake house, and schedule batch or streaming jobs in fabric.

Explore data mesh architecture as a federated data model where data is owned and governed by each business unit, moving from centralized storage to domain-driven governance in fabric.

Learn to create and manage domains in Fabric to enable data mesh architecture, assign domain admins, and organize workspaces with governed access via the Data Hub.

Learn how to migrate from Azure Synapse Analytics or Azure Data Factory to Microsoft Fabric, including lift-and-shift options, manual creation needs, and running existing Synapse or ADF pipelines from Fabric.

Discover how to migrate Synapse notebooks to Microsoft Fabric, compare manual export/import with an API-driven automated approach, and scale from a single notebook to 50-plus notebooks.

Learn how to migrate synapse notebooks to the fabric workspace using the fabric api, with prerequisites including a fabric workspace, lakehouse, and a service principal.

Create a service principal, grant Synapse access, export notebooks to the fabric lake house, then import them as prefixed Mig notebooks for a complete Synapse-to-fabric migration.

Assess the migration of synapse and data factory pipelines to fabric, noting limitations that require manual recreation and the shift from link services to connections.

Explore data migration options from Azure Data Lake Gen2 to Fabric OneLake, including shortcuts, fast copy, easy copy (AzCopy), pipelines copy activity, and Azure Storage Explorer.

Migrate ADLS data to one lake in Fabric using Storage Explorer by signing in, attaching the one lake resource, granting contributor access, and copying ADLS Gen2 containers.

Explore how to use the capacity metrics app in Microsoft Fabric to monitor and optimize resource usage, manage capacity efficiently for high-demand workloads, and ensure environments run smoothly and efficiently.

Learn to install and configure the Capacity Metrics app in Microsoft Fabric to monitor capacity usage, track resource consumption, and gain insights into compute and storage metrics.

Explore the capacity metrics UI in the fabric project, featuring compute and storage tabs, hourly multi-metric ribbon charts, a time-based drillable matrix, and item-level usage insights.

Understand capacity units consumption in a 32nd window, including interactive vs background workloads, throttling, and smoothing, with practical interpretation of capacity unit seconds and utilization.

Examine throttling and smoothing in Microsoft Fabric, and learn how capacity balance manages resources. Know how smoothing shifts usage from peak to idle, preventing overuse, interactive delay, and background rejection.

Understand how fabric throttling stages handle overage beyond 100% using a ten-minute threshold and 24-hour smoothing. Note how interactive workloads transition to the next stage, interactive delay.

Explore throttling stages in Microsoft Fabric, from interactive delay to interactive rejection and background rejection, as capacity usage above 100% over ten minutes impacts notebooks and SQL queries.

Explore throttling stages from overage protection to background rejection, including ten-minute compensation, interactive delay, and interactive rejection, and plan capacity accordingly.

Understand how to track overages in Fabric using burn down and carry forward metrics within a 32nd period window, including billing operations and cumulative carry forward.

Track system events in fabric to monitor pause and resume capacity tied to Azure Portal licenses, and assess how often you can pause unused capacity.

Explore the matrix visual to filter 14-day window by item kind (spark notebook, lake house, warehouse, pipeline), add columns like in progress, successful, and performance delta, and identify notebook utilization.

Explore the Synapse data warehouse in Microsoft Fabric as a platform for enterprise scale analytics, and compare data loading, lake house versus data warehouse, plus cloning time travel.

Create and manage a Fabric warehouse that stores structured data in tables, enables transactional SQL, DDL/DML, and schema changes, with a default semantic model for Power BI datasets.

Compare Fabric's warehouse and sql analytics endpoint: warehouse enables full t-sql with dml and ddl on delta parquet data; sql analytics endpoint provides read-only tsql access to lakehouse data.

Create a table in the Fabric warehouse and review key limitations, including case-sensitive object and column names, and the lack of primary key and unique constraints.

Learn four data ingestion methods to load data into a warehouse, including copy into, pipelines, data flow gen2, and cross-database ingestion from lake house and fabric warehouses.

Copy data from Azure Data Lake Gen2 into the warehouse using the copy into command. Authenticate via role-based access control and configure the source path and parquet files.

Ingest data into the fabric warehouse using a pipeline from an on-prem sql server, with a staging area in blob or adls storage and an on-prem data gateway.

Use dataflow gen2 to copy data from on-prem sql server to the warehouse, applying transformations and adding a new column with today’s date, then publish and refresh.

Access lake house data from the warehouse and vice versa using sql analytics endpoint in Microsoft Fabric, enabling cross-database queries between lake house and warehouse tables.

Explore cross database ingestion to move data from lake house to warehouse using create table as select, insert into, and select into methods.

Compare lakehouse and warehouse to decide when to use each, based on data type and preferred tooling, noting delta format stores all data and performance remains similar.

Explore medallion architecture patterns using lake house and warehouse designs, from landing zone ingestion to bronze, silver, and gold layers, with spark transformations and sql-ready reporting.

This lecture demonstrates updating data between lakehouse and warehouse in Fabric, noting the SQL analytics endpoint is read-only. Use a lakehouse shortcut for PySpark, but cross updates aren’t supported.

Explains SQL query as a session in fabric, showing that the fabric SQL window treats each statement as a separate session, preventing temporary tables, unlike SSMs with persistent context.

Master the zero copy clone feature in fabric to clone tables by metadata only, enabling testing across schemas without affecting original data in the one lake.

Learn time travel in the warehouse with table clone, cloning from the current state or a previous point in time up to seven days using a UTC timestamp.

Explore zero copy clones for safe development and testing in production-like environments. Note the limitations: no cross-warehouse cloning, no workspace cloning, read-only sql analytics endpoint, and seven-day retention.

Clone tables through the UI by selecting single or multiple tables, choosing the current or past timestamps, and viewing the SQL statement equivalents—achieving low-code, no-code cloning and faster schema replication.

Explore Query Insights in the warehouse to monitor and optimize performance, viewing historical and aggregated data for execute request history, frequently run queries, and long running queries.