Microsoft Fabric Data Engineering: 10+ Real-Time Projects

Microsoft Fabric unifies data engineering and analytics in a single SaaS platform, bringing Power BI, Azure Data Factory, Spark, data lake, and lake house together for end-to-end analytics.

Create an azure free account by searching for azure free, entering personal and company details, solving a puzzle, and verifying with a credit card and OTP on portal.azure.com.

Learn how to create a Microsoft Fabric account and start a 60-day free trial with an organization email, or use Azure free credits to set up Fabric capacity.

Explore fabric pricing and capacity units, showing how an Azure free account for 30 days with a $200 credit minimizes costs by pausing unused capacity to about 2.88 K.

Explore the Microsoft Fabric hierarchy by defining the tenant, workspace, capacity, and items, including Power BI Premium per user, and learn how capacity types power notebooks, pipelines, and SQL warehouses.

navigate the microsoft fabric overview, manage workspaces, and explore data catalogs and real-time analytics, while reviewing workloads like data engineering, data science, data warehouse, and power bi.

A workspace in Microsoft Fabric is a collaborative container for data assets like lakehouse, data pipelines, notebooks, warehouse, and Power BI reports, created using an Azure fabric capacity.

Discover how one lake unifies analytics data on fabric, built on Azure Data Lake Storage Gen2, storing delta parquet data for all workloads.

Explore how lakehouse blends data lake and data warehouse in fabric, storing unstructured files and structured delta tables in one place, with SQL and PySpark queries and Power BI visualization.

Create a fabric lakehouse inside a workspace, upload files to the file section, and optionally create a table. Access the semantic model, open notebook, and SQL endpoints for analytics.

Explore how the One Lake File Explorer integrates Windows with Microsoft Fabric, enabling you to access, upload, view, refresh, and delete lakehouse files from both desktop and Fabric workspaces.

Create a table from an uploaded CSV in a lakehouse with Microsoft Fabric. Learn header-based column mapping, parquet and delta table creation, SQL queries, and managed versus unmanaged table types.

Explore shortcuts in Microsoft Fabric, virtual pointers that reference data in other locations without copying, enabling cross-workspace access; distinguish internal and external shortcuts and their benefits.

Create and link data across multiple fabric workspaces by building lake houses, uploading files, and generating shortcuts for files and tables to access cross-workspace data.

Create external shortcuts in fabric to access data stored in Azure Data Lake Storage without moving it, enabling cross-lake analytics from ADLS to fabric.

Learn how fabric enables caching for external shortcuts to data outside the one lake, creating a temporary cached copy in the one lake to boost read performance and save costs.

Learn how to use sql endpoints in microsoft fabric to query lake house data with t-sql, analyze delta table data, and share results with Power BI via sql connection string.

Explore fabric data factory, a next-gen data orchestration service for ETL and ELT, enabling pipelines that ingest, transform, and load data from multiple sources into a lake house.

Learn to create a data factory in Microsoft Fabric by building a workspace, naming a factory or pipeline, using a blank canvas, and adding copy data with validation and run.

Learn to copy data from an Azure SQL DB to a lakehouse using a copy data activity, exporting as parquet or JSON via a Data Factory pipeline.

Use the delete activity to remove existing data in a branch folder before daily copy from S3, ensuring only the latest data remains.

Use the get metadata activity in fabric to extract file metadata, identify files in a container, view last modified dates, and filter by file type or name.

filter activity in fabric data factory to select csv files from blob storage by using get metadata and a condition that the file name contains csv.

Learn how the lookup activity in Fabric retrieves data from tables, files, or queries, returning an array that informs downstream actions like deletions or copies in a pipeline.

Learn how parameters in fabric data factory make pipelines dynamic and reusable by passing file names at runtime to copy employee.csv from ADLs to lake house.

Use the for each activity in Fabric to iterate files, read metadata, filter csv files in adls, and delete them with a delete activity.

Apply the if condition activity in Microsoft Fabric to route files by name. Read metadata from ADLs, loop with for each, delete EMP files, and copy others to the lakehouse.

learn how to set up notifications in fabric using office 365 outlook or teams, sign in to your organization account, and receive alerts when a pipeline fails.

Learn to use the invoke pipeline activity in Fabric to call subsequent pipelines within a data factory workspace, set up a connection, select pipelines, pass parameters, and orchestrate multi-pipeline flows.

Learn how to schedule a pipeline in Fabric on hourly, daily, weekly, or monthly intervals, set the start date and time zone, and automate runs.

Learn to monitor your Fabric data pipelines by viewing run history, filtering to data pipelines, and inspecting failure details, durations, and error messages to diagnose issues.

Use the wait activity in Microsoft Fabric to pause pipelines between steps and control timing. Implement retries after fetch failures by waiting before retrying.

Explore data flow Gen2 in Microsoft Fabric to clean, transform, and load data with Power Query via a low-code visual interface from ADLS Gen2, SQL Server, Excel, or Dataverse.

Connect sources in fabric using dataflow Gen2 by uploading files to a lakehouse and transforming data with no-code steps like removing a column, removing duplicates, sorting, and changing data types.

Transform string data in Fabric by applying uppercase, lowercase, capitalize each word, trim whitespace, add prefix or suffix, split by delimiter, replace values, and rename columns.

Learn to perform mathematical transformations in fabric dataflow, computing sum, minimum, maximum, and average, counting distinct values, and applying operations such as add, subtract, multiply, divide, percent, and square root.

Explore data flow gen2's diagram view in fabric to visualize applied transformations, from column changes to sorting and grouping by city with the sum of sales.

Perform a join in Microsoft Fabric by using merge queries as new to combine two data sources, selecting customer id from each table and applying left or inner joins.

Learn how to load transformed data into a final destination table in the lakehouse using data factory, configure mappings, publish and run the data flow, and schedule via pipelines.

Discover spark in fabric as a general-purpose in-memory computation engine that accelerates data engineering tasks, enabling fast cleaning, transformation, and analytics with Spark SQL and Spark MLlib.

Explore pools in Microsoft Fabric, comparing starter and custom pools, their setup times, performance, and billing for development versus production workloads.

Create and run your first spark notebook in fabric's data engineering workspace, writing PySpark, SQL, or R code, managing pools and clusters, and using notebooks, lake house, and environments.

Create a data frame in PySpark by reading a csv from a lakehouse. Use header true and infer schema to auto-detect types, and explore read options with spark.read.csv and spark.read.format.

Learn how to use PySpark to read a data frame, display data, select columns (id, sales, product name), and limit or drop columns to view a focused dataset.

Learn to use with column in PySpark to cast the quantity column from string to integer. Create new columns like quantity one and country, update values, and rename zip mode.

Learn how to sort PySpark dataframes by single and multiple columns using df.sort and df.orderBy, with ascending and descending options on fields like customer name, sales, and profit.

Master group by in PySpark by aggregating with sum, min, max, and count to analyze segment and city profit in a dataframe, displaying segment-wise results.

Explore filtering data in PySpark on fabric using df.filter with country and segment conditions, and pattern matching on customer name starting with capital C.

Handle nulls in pyspark on fabric by reading a csv into a dataframe, filtering with is null or is not null, and replacing missing values with df.na.fill for specific columns.

Convert a PySpark data frame into a temporary view, then perform transformations with spark sql or sql, and optionally convert back to a data frame within the session.

Learn to create in-notebook visualizations from a Spark data frame by turning it into a temp view and using SQL queries for bar, line, and pie charts.

Write a cleaned PySpark dataframe to a delta lake, choosing append mode and saving as a delta table, while creating managed or external tables in a lakehouse for scalable analytics.

Learn to use notebook utils in fabric notebooks to list, copy, move files with fs, cp, mv, and run or exit notebooks from one notebook to another.

Create and manage environments in fabric to tailor pools, spark versions, and libraries for specific notebooks. Notebooks inherit workspace settings, or switch to a custom environment.

Learn to schedule a notebook in Fabric using direct scheduling or data pipeline, choosing hourly, daily, weekly, or monthly runs, and monitor jobs with spark application details and logs.

Create a dedicated data warehouse in Microsoft Fabric to centralize clean, structured data for analytics and reporting, using ETL and star or snowflake schemas for efficient, historical insights.

learn to create and manage a data warehouse in fabric within a workspace, write SQL queries, create tables, insert data, and explore results with visualizations and Power BI.

Learn how to load data into a data warehouse using three methods: a data factory pipeline or a data flow, copy into, and create table as select.

Use copy into to load data from adls to the data warehouse, creating the gold schema and employee table, with csv format, sas token, and varchar casting.

Use CTAS to load filtered consumer data from a destination lakehouse into a gold schema table, creating the new table via save as table.

Understand views as virtual tables created from a SQL query that do not store data, and learn to create reusable views to simplify complex joins and enable BI integration.

Learn to create and parameterize stored procedures in T-SQL to reuse complex queries, call them at runtime with different country values using execute, and contrast with views.

Explore the new visual query in the DWH fabric, transforming the employee table with filters, value replacements, and view creation, while seeing the generated sql.

Explore downloading a sql database project to see table schemas and the associated queries, including employee and consumer tables within dbo and gold schemas in a data warehouse.

Learn how to run T-SQL directly in a notebook by connecting to your data warehouse, selecting T-SQL, adding the warehouse, and executing queries like selecting from consumer.

Learn how Power BI visualizes data in Microsoft Fabric, connecting lake house, warehouses, and SQL endpoints to build reports and visualizations using Power BI Desktop and DAX.

Power BI, a Microsoft business intelligence tool, connects to diverse data sources, cleans data, and creates interactive dashboards and visual reports for real-time insights shared with stakeholders.

Download and install Microsoft Power BI Desktop from the official site, selecting the 64-bit version, then open the app to practice after installation.

Explore Power BI desktop by learning how to connect to multiple data sources, navigate the report, table, model, and DAX views, and create visualizations with the visualization tab.

Load an Excel workbook into Power BI Desktop by selecting four sheets (customer, order, people, return), adjust data types, view relationships in the data model, and begin visualizations.

Set up a Power BI dashboard page, adjust the size and background, and add a global header with a text box; import a logo linked to a web URL.

Learn to create and customize card visuals in a dashboard to display total sales, total profit, number of customers, and total unique orders using sum and distinct counts.

Learn to build a year-over-year bar chart in Power BI using a date hierarchy and sales data, including formatting, colors per year, data labels, and interactive filtering.

Explore how to build a line chart in Power BI desktop to display monthly sales and profit trends, format axes and data labels, and highlight seasonality across years.

Create a line and stacked column chart to display market-wise sales as bars and profit as a line in a Power BI dashboard, enabling interactive market filtering.

Visualize quarterly sales with a pie chart in Power BI desktop, using quarter as the legend and the sum of sales as the value, applying formatting and format painter.

Learn how to use a Power BI slicer to create dynamic visuals that filter bar, line, and pie charts by segment or category, with dropdowns, select all, and formatting options.

Learn to create static visuals in Power BI by configuring edit interactions to keep total sales and total profit constant on a dashboard.

Add a Power BI sales KPI to compare yearly sales against a 5 million target and quantify distance from the goal.

Learn to add a gauss chart to a dashboard to show how far the total profit (1.47 million) is from the 5 million target, using KPI context.

Create a one-click clear all slicers button to return to the default view, including configuring the button text and style, and using the Ctrl+Enter shortcut to reset all filters.

Explore creating map visuals in Power BI, including bubble and field maps, to display country profit with a red-to-green gradient and sales under or over one lakh.

Explore how the q&a chart in Power BI generates visuals from natural-language questions. Ask prompts like sum of sales by quarter to get a pie chart or bar chart.

Discover how to use tooltips in Power BI to display extra chart information on hover, including sum of sales, sum of profit, and count of order ID.

Learn how to apply Power BI filters by country, using page, all pages, and visual filters, with basic and advanced options and top-n insights.

Learn how to drill up and drill down in Power BI using a date hierarchy to analyze sales across year, quarter, month, and day.

Learn how to model data in Power BI by defining data types and establishing relationships between tables such as customer, order, and product, including cardinalities and cross filter direction.

Explore DAX, a data analysis expression language used in Power BI, Power Pivot, and the SS tabular model, to create new calculated columns and measures.

Explore the DAX format function to format columns, from adding thousand separators to displaying currency, percent, and date values, using practical examples on a Global Superstore data set.

Apply DAX date diff and if conditions to classify orders as on time or delayed, using order date and ship date, and visualize results with a pie chart.

Derive the day of week from the order date using format, then apply a DAX switch to label each date as weekend or weekday via a new column.

Learn how to use a DAX variable to classify orders as weekend or weekdays, replacing multiple derived columns with a single, efficient expression.

Learn to use the DAX calculate function to compute the sum of sales filtered by order date from April 1, 2019 to April 15, 2019 in Power BI.

Group data in Power BI by creating a region group column that maps regions into north, central, east, and south, then visualize sales by the region in a pie chart.

discover how measures in Power BI use DAX formulas to perform aggregation, understand implicit versus explicit measures, and learn when to use measures instead of columns for efficient data analysis.

Learn to build Power BI measures that compute average sales per customer from total sales divided by distinct orders, and filter to show the top ten customers.

Explore how to use the rankx function in Power BI to rank countries by total sales, create measures, and filter ranges such as top 4 to top 9.

Learn to clean and transform data in Power BI with the Power Query editor, fixing headers and data types, handling nulls and duplicates, and applying joins before visualization.

Master the Power Query editor to transform data and set the first row as header. Derive year, month, and quarter from order dates, then apply changes to Power BI desktop.

Master text modification in Power Query using extract, format, and parse to transform a text column, including length, first or last characters, delimiter splits, and case changes.

Master the Power Query editor to add and reorder index columns, duplicate columns, and assess column quality with profile, distribution, and query dependency.

Learn to handle null values in Power Query by removing blank rows, and replacing nulls with a value or Na after converting numeric columns to text when needed.

Learn how to create conditional and custom columns in Power Query using if and else with and/or logic to derive weekend or weekdays and financial quarters.

Explore Power Query merge operations by joining the order and customer tables on customer ID, using inner, left outer, right outer, and full outer joins, and contrast with append.

Learn how to union two tables in power query by using append query. Ensure columns and data types match to successfully combine data from both tables.

Execute an end-to-end Azure data engineering project with Microsoft Fabric to build a unified customer 360 dataset for an e-commerce company, using Data Factory, PySpark, Lake House, and Power BI.

Set up an Azure ADLS storage account with hierarchical namespace, create a lakehouse with silver and gold folders, and run a PySpark notebook to clean and store data.

create a new fabric workspace and lake house, implement a medallion architecture with branch, silver, and gold layers, and build a data pipeline to convert adls csv to parquet.

Create a data pipeline that reads files from Azure Data Lake Gen2, uses get metadata, and for each file copies them into the lake house as parquet.

Read messy lakehouse data with PySpark, clean it into bronze and silver datasets, then build a gold customer 360 and Power BI report.

Explore an end-to-end Azure data engineering project on Microsoft Fabric, applying medallion architecture to create a unified retail data view from CSV, Excel, and JSON sources and clean with PySpark.

Learn to create an Azure Data Lake Storage Gen2, build a medallion lakehouse in Fabric, and copy JSON, CSV, and Excel to Parquet in bronze, silver, and gold.

Learn to build a lakehouse data pipeline with PySpark to read parquet data, clean it, and create bronze, silver, and gold delta tables for analytics and Power BI reporting.

Explore an end-to-end data migration project in Azure data engineering using Microsoft Fabric, building a complete data pipeline driven by a business scenario and the solutioning steps.

Demonstrates migrating large data from ADLs to fabric lakehouse and fabric data warehouse using copy data and pipelines, with a file tracker to skip already loaded files.

Set up an end-to-end data pipeline in Azure, creating ADLS storage, a lakehouse, and a data warehouse, then use get metadata, lookup, filter, and for each to copy data.