Microsoft Fabric - The Complete Guide

Discover how one lake serves as a centralized, invisible data repository with shortcuts to external files, enabling seamless access in fabric through workspaces and lake houses.

Create workspaces to house lakehouses, reports, dashboards, and data flows, assign permissions, and share lakehouses across workspaces for Power BI and data engineering tasks.

Explore lake houses as a single, flexible workspace for storing delta tables and both structured and unstructured data, with a read-only SQL endpoint and easy tool integration.

This lecture introduces the unified fabric developer experience, replacing separate workloads with one integrated platform to create items—from data pipelines to reports and dashboards—within a workspace, including Power BI integration.

Explore the Fabric and Power BI views to create items and reports in your workspace, manage access with roles, and adjust settings for the lake house in the next lecture.

Create your first lake house in the workspace and open its SQL endpoint. Explore the data view, tables, and files, and connect with external tools via the SQL connection string.

Learn to load data into a lake house by uploading csv files, organizing them in subfolders, previewing content, and converting files into tables for loading into existing or new tables.

Convert a csv file into a delta table for analytics, fix header issues with underscores, and load data into a new table with a preview of column types.

Explore delta tables built from parquet files with a log transactions folder, view data and delta history, and understand appending new data and updating json logs.

Create and run dataflow Gen 2, a no-code ETL tool using Power Query to load JSON data into a lakehouse delta table. Explore connections, transformations, and scheduling to automate refreshes.

install the one lake file explorer, sign in, and access your fabric workspace from windows explorer to drag and drop files for lakehouse cloud sync.

Explore the SQL analytics endpoint in Microsoft Fabric, connect with SQL connection string, run queries on the sales table, and save as views or create visual queries for easier analysis.

Create and save visual queries in the SQL analytics endpoint using a Power Query–like interface to rename, filter, join, and transform tables, then run SQL.

Create shortcuts in fabric to reference data from external sources like S3 without copying it. Access the shortcut as a delta table with full querying and semantic modeling.

Learn to manage access in a Fabric workspace by assigning roles, such as admin, member, contributor, or viewer, and sharing items like a lake house with selected users.

Explore how semantic models underpin Power BI reports by defining table relationships, loading data into a lake house, and establishing one-to-many links between product ID and sales to enable visuals.

Create a new semantic model in the lake house for Power BI using direct lake connectivity to avoid data copies and support multiple use cases.

Create measures in the semantic model to perform virtual calculations that are reusable across reports, using a simple sum-based discount example in a sales table.

Connect Power BI Desktop to Fabric lake house semantic models for live data visualization, sign in with two-factor authentication, and publish reports to a Fabric workspace.

Connect the SQL endpoint directly in Power BI using the connection string, sign in with two factor authentication, and load tables in import mode for transformation and modeling.

Explore auto create report in fabric to generate Power BI reports from a semantic model, adjust visuals, and publish polished reports via app distribution.

Learn how to create apps in Power BI within Microsoft Fabric, distribute reports to end users, manage content and audiences, publish apps, and share access links.

Update the app by editing underlying Power BI reports, publish changes, and adjust content, audience, color, and logo, and when needed unpublish or delete the app in Fabric.

Learn how Microsoft Fabric uses data pipelines for orchestration and data flow gen2 for transformation, with Power Query editor, connecting sources, shaping data, and loading to lake house or warehouse.

Create a data pipeline in fabric using the copy data activity for ELT, connect to parquet retail data, and load into a lakehouse table with mappings.

Add data flow gen 2 to a Microsoft Fabric data pipeline to transform data in the Power Query editor, connect to a lakehouse, publish, and schedule the workflow.

Disable staging in data flows to boost performance; reduce runtime from 3.5 hours to about 1 hour for 50 million rows.

Learn to run a data pipeline on demand or schedule it with minute, hourly, daily, or weekly intervals, view run history, and apply changes.

Learn to troubleshoot and monitor data pipelines using run history, monitoring hub, and gantt views; inspect per-activity details, input/output JSON, data read/write, and cancel or manually trigger runs.

Explain what a data warehouse is, compare it with a lake house, and highlight Fabric’s fully managed, write-enabled warehouse versus the lake house’s file support and read-only SQL endpoint.

Create your first data warehouse in Microsoft Fabric, explore its features from data engineering to a SQL perspective, use delta parquet format, and build tables with T-SQL and data pipelines.

Compare Fabric data warehouse with lake house read-only SQL endpoint, then demonstrate creating tables using T-SQL, with not null constraints, and observe table creation and basic queries.

Insert data with insert into, specify columns and values, verify via select, then update rows with a where clause, and sort results by order by.

Modify schema by adding a date id column and changing a column's data type with alter table, or use create table as select to preserve data when alter is unsupported.

This lecture demonstrates calculating a fixed date_id, linking the date table to public holidays with a 1-to-1 key, and building a semantic model using the copy into command.

Learn to use the copy into command to ingest data from Azure blob storage into a data warehouse, creating and populating the dimension city and fact sale tables.

Demonstrates building a data pipeline to copy a parquet file from Azure Blob Storage into a warehouse table, including configuring copy activity, connections, and validation.

Reference SQL endpoints from other lakehouses without copying data by adding them to the explorer and exposing them through a view in your warehouse for the semantic model.

Explore cloning tables in Microsoft Fabric: use the clone command to create reference-only tables that don’t copy data, enabling fast, storage-efficient updates across clones and originals.

Create a semantic model in the warehouse to power Power BI reports, then connect via the SQL endpoint and reuse the model with selected tables and relationships.

Explore Apache Spark in Fabric as a distributed in-memory compute framework for data engineering and data science, with notebooks and multi-language support for large-scale batch and real-time processing.

Create a spark data frame in fabric notebooks by defining a python data list and a manual schema, then display results and explore loading data from files and lakehouses.

Navigate notebooks by executing cells interactively, manage variables and outputs, and control cell states with freeze and lock options while using markdown to organize content.

Upload files or reference data from anywhere, then read csv and json with pandas in notebook, with spark for larger lake house files.

Connect and read data from a lakehouse using spark, reading csv and json with drag-and-drop and spark.read, while noting that by default spark infers all types as strings.

Define data schemas by manually specifying types or automatically inferring them in pandas and spark, ensuring data integrity in large lakehouse datasets.

Apply data filtering with the filter method on a data frame, using single and/or multiple conditions, and between timestamps to select 2023 January high-value transactions, then display results.

Modify a data frame schema with Spark by casting types, renaming columns, and dropping or adding columns. Use withColumn, withColumnRenamed, printSchema, and quick before-after checks to validate changes.

Add a new column to a dataframe using Spark's lit function, converting a literal value into a column, and explore simple SQL-like data manipulation ahead of aggregation.

Learn to aggregate and group data with PySpark SQL, applying average and sum, aliasing results, and grouping by customer to compute total sales.

Import Python functions into PySpark, use the SQL round function to two decimals, rename results for product-level insights, and plan writing data back to the lakehouse.

Learn to join data frames in spark with the join method to create a data frame by matching columns, using inner, left, right, and outer joins on sales and products.

Write data frames to lakehouse as csv, json, or parquet files, using overwrite or append modes, and store into delta tables for semantic models and Power BI visualization.

write back data as delta tables in a lakehouse with spark sql, save as a table, and switch between overwrite, append, or error modes, enabling visualizations in Power BI.

Learn to run sql queries in spark sql within notebooks, perform filters, joins, and aggregations, and switch between spark sql and spark python to work with tables and data frames.

Learn how to load a CSV into a data frame, create temporary views for sales and products, and query them with SQL in a notebook, enabling Spark and SQL integration.

Explore when to use SQL vs PySpark, and combine them with temporary views to filter and transform sales data using year and month extractions, while handling column naming nuances.

Learn to schedule notebooks for interactive exploration, visualizations, and machine learning, and link them to data pipelines, with options for hourly, daily, weekly runs and spark jobs.

Learn how to integrate notebooks into data pipelines, configure pipeline activities, set dependencies, and orchestrate workflows from notebook to spark job definition for scalable production.

Explore spark settings in your workspace, configure pools and node sizes, enable auto scaling, and optimize spark workloads for data science, notebooks, and machine learning experiments.

Explore spark job definitions for production, including creating, scheduling, and running jobs that read csv data, transform it, and write parquet files to a lakehouse.

Learn to schedule and monitor spark job definitions, configure retry policies and environments, and analyze run metrics across spark jobs and notebooks.

Explore real time analytics in fabric, focusing on the real time hub and how streams and tables in the workspace enable end-to-end data capture, processing, and visualization.

Explore the main building blocks of real-time analytics in fabric, including the event house, Kql databases and tables, and event streams, and how the real-time hub organizes streaming data.

Create a KQL database inside a Fabric event house, load a sample stocks analytics dataset, explore the stocks table, and run basic queries to learn KQL syntax and query sets.

Explore the basics of KQL in Microsoft Fabric by building piped data operations from a table source, selecting columns with project, taking 100 rows, and considering performance with columnar storage.

Learn how to filter and sort data using KCL: apply where conditions, project specific columns (date, close, volume), and order by date descending to efficiently query and view results.

Aggregate data with summarize to compute counts, max close, and averages. Group by ticker and date, bin with week of year, and prepare visuals in queries.

Explore visualizing data with kcl using render to create pie, bar, column, and time charts from time-series data, with where filters, for interactive, high-performance insights in Fabric.

Upload a csv to create a new table, adjust the schema, and ingest data; then join the company info with the stocks table using kql.

Learn to join data in Kql by combining stocks and company tables, using inner and full outer joins, projecting specific columns, limiting results, and preparing for event streams in Fabric.

Learn to create and configure event streams in a Microsoft Fabric workspace, ingest real-time data via a visual editor, apply transformations, and route results to Kql databases or other destinations.

Route a new event stream into a KQL database by configuring an event house destination, creating a new table, selecting JSON input, publishing the live data flow, and applying transformations.

Edit the live event stream to apply transformation events after data capture, modify fields and types, and publish enriched data from source to destination in real time.

Connect a KQL database to Power BI to visualize streaming data with live direct query or import. Use a semantic model optimized for Power BI to enable lake house notebooks.

Learn to ingest stream data into a lake house using a one lake shortcut from the event house, enabling table availability and optional destination for the semantic model.

Build an end-to-end fabric-based classification model from data ingestion to applying to new data, using notebooks, experiments, and mlflow to track, register, and reuse the model.

Import and load a notebook, set up the workspace, and load the data to perform exploratory data analysis with data wrangler, then train and compare models for customer churn prediction.

Learn to load data into a spark data frame, cache for performance, convert to a pandas data frame, and store results as a lake house table for exploratory data analysis.

Explore the data with Data Wrangler, check for missing values, and apply cleansing steps like dropping columns, removing duplicates, and one-hot encoding to prepare for modeling.

Explore exploratory data analysis with data wrangler, identify target and variable types, assess distributions and outliers, perform feature engineering, and build and evaluate models.

Explore feature impact analysis to identify which attributes influence the target variable, using basic measures and a clustered bar chart, guiding subsequent feature engineering for the machine learning model.

Explore feature engineering to improve model performance by creating features, such as the ratio between Tanger and H, credit score buckets, and encoding categoricals, then compare LightGBM and random forest.

Connect clean delta table data to Power BI for exploratory analysis and reporting, build histograms from age, and summarize churn by country for model training.

Set up experiments with MLflow auto logging to train and compare models like LightGBM and random forest on prepared, split data, then evaluate results to select the best model.

Split the data into train and test sets, using a 20% test split with a reproducible seed, oversample the minority class to balance the imbalanced data before training models.

Train a random forest classifier within the experiment using balanced data. Track metrics, generate a report and confusion matrix, and prepare for comparing runs in the next lecture.

Register models in an experiment with auto logging, then evaluate and compare runs using accuracy, precision, recall, and F1 score, aided by a confusion matrix.

Explore the precision recall curve and ROC curve, and learn how thresholds shape precision, recall, true positive rate, and false positives to compare models for deployment.

Learn to compare model runs in Fabric, explore metrics and hyperparameters, evaluate random forest and lightgbm, visualize results, and select and save the best model for deployment.

Save a run as an ml model to reuse it across data tables, creating a model item in the workspace that you can apply to new data in production.

Apply the trained model via wizard or code, map input data to the model, output predictions to a lake house table, and schedule notebooks for end-to-end automation.

Explore fabric activator, a real-time intelligence component that detects events such as threshold crossings and data changes, and automatically triggers actions like alerts or emails, with Power BI examples.

Create reflex from Power BI data by configuring fabric activator to monitor real-time metrics, set alerts on visuals, and auto send email or Teams notifications when conditions change.

Feed real-time data from an event stream into a reflex to create streaming objects such as street and bike counts, and configure triggers to act on this data.

Learn to build and monitor data triggers from objects in Microsoft Fabric, using event streams, predefined properties, and reflexes to activate alerts and test email notifications.