Microsoft Fabric: End to End Data Engineering Project

Name: Microsoft Fabric: End to End Data Engineering Project
Rating: 4.5 (524 reviews)

Build Bing News Data Analytics platform using different Data Engineering components of Microsoft Fabric [DP-600][DP-203]

Created byMr. K Talks Tech

Last updated 8/2025

English

English [Auto],French [Auto],

What you'll learn

You will learn to use Microsoft Fabric for building a Bing News Data Analytics platform, enabling seamless integration with Azure Data Engineering components
You will learn the process of ingesting data from external sources, specifically utilizing Bing API, using Data Factory.
You will learn to perform data transformation techniques to shape and refine raw JSON data into curated Delta Tables using Synapse Data Engineering component
You will learn how to perform sentiment analysis using Synapse Data Science component
You will learn how to orchestrate data workflows with Data Factory pipelines.
You will learn how to perform Incremental Load using spark notebooks.
You will learn how to visualize data effectively using Power BI.
You will learn how to configure alerts within Power BI visuals with Data Activator.

Course content

2 sections • 15 lectures • 3h 42m total length

Introduction to Microsoft Fabric16:33
Explore Microsoft Fabric, a unified analytics platform that integrates Data Factory, Synapse components, Power BI, and one lake storage, driven by serverless compute for data engineering, science, and analysis.
Create and Enable Microsoft Fabric12:17
Enable Microsoft Fabric in a Power BI workspace, register fabric in Azure, and choose fabric capacity, premium, or a 60-day trial to unlock fabric components.

Project Overview14:20
Demonstrates end-to-end azure data engineering project using Microsoft Fabric, ingesting Bing news via Bing API, transforming to delta tables, running sentiment analysis, and building a Power BI dashboard with alerts.
Environment Setup12:56
Create a dedicated resource group and the Bing Search API in the Azure portal to enable data ingestion, then set up a Power BI Fabric workspace and lake database.
Read Me (*Important*) - Environment Setup1:12
Data Ingestion using Data Factory24:16
Connect to the Bing API using data factory, configure a copy data pipeline, and ingest the latest news into the Lakehouse database as a JSON file for downstream transformation.
Read Me (*Important*)- Data Ingestion using Data Factory1:30
Data Transformation using Synapse Data Engineering34:58
Perform data transformation with synapse data engineering by reading a json file, using a spark notebook to explode and parse json objects, and write a delta table with date formatting.
Incremental Load using Spark Notebook22:40
Learn to implement incremental load with Spark notebook by using delta table merge to load only new data into the AES database, choosing type one merge over overwrite or append.
Sentiment Analysis using Synapse Machine Learning15:54
Perform sentiment analysis in Microsoft Fabric with Synapse ML, using a notebook workflow to transform raw data into a lakehouse delta table and apply a type one merge.
Building Reports using Power BI25:30
Build Power BI reports by connecting to the Lakehouse with a semantic model and using the TBL sentiment analysis to create a news dashboard with latest data and sentiment measures.
Read Me (*Important*)1:00
Building Pipelines using Data Factory for Orchestration20:50
Build end-to-end pipelines with Data Factory to orchestrate data ingestion, transformation, and sentiment analysis for news articles, loading into the lakehouse and powering Power BI reports.
Setting up Alerts using Data Activator10:14
Configure end-to-end alerts in Power BI using data activator by creating reflex items, linking visuals like the positive sentiment card, setting threshold conditions, and routing alerts to Microsoft Teams.
End to End Pipeline Testing7:52
Perform an end-to-end data pipeline test in Microsoft Fabric, ingesting news via Bing API, processing with Synapse to a Delta table, and triggering Teams alerts via Data Activator.

Requirements

Basic Programming Skills
Watch the two Pre-requisite videos

Description

In this project, we will build a Bing News Data Analytics platform!

This would be a complete end to end Azure Data Engineering project that's done using Microsoft Fabric. We'll pull raw data from Bing API, transform the raw data to clean data with Synapse Data Engineering, analyze sentiment with Synapse Data Science, set up workflows with Data Factory, make cool reports with Power BI, set alerts with Data Activator, and test everything well. Let's get started on Bing News Analytics!

The Topics covered in this Project are,

1. Data Ingestion from Bing API using Data Factory: Learn how to seamlessly pull in data from external sources, setting the foundation for your analytics project.

2. Data Transformation using Synapse Data Engineering: Dive into the process of shaping and refining your raw JSON data to a curated Delta Table, including techniques like incremental loading to keep your processes efficient.

3. Sentiment Analysis using Synapse Data Science: Uncover insights hidden within the news description by predicting the sentiment of the news classified as Positive, Negative or Neutral.

4. Orchestration using Data Factory via pipelines: Discover the art of orchestrating your data workflows, ensuring smooth and efficient operations.

5. Data Reporting using Power BI: Visualize your data in a compelling and actionable manner, empowering stakeholders with valuable insights.

6. Configuring Alerts using the Data Activator: Stay ahead of potential issues by setting up alerts and notifications within your Power BI visuals using a new tool called Data Activator.

7. End to End Pipeline Testing: The complete flow will be tested right from the data ingestion to the data transformation and until the report gets updated with the incoming new data to Validate the integrity and performance of your pipelines, ensuring reliability and accuracy.

This project revolves around Bing News Data Analytics, a practical application that involves ingesting news data daily and generating insightful reports. By walking through each step in a simplified manner, I aim to make Azure Data Engineering accessible to all enthusiasts, regardless of their background.

Pre-requisites

You don't need to have any skills to do this project. Even if you are an absolute beginner, if you follow the entire course, you will be able to implement this project, The only pre-requisite for this project is, you need to two introductory videos of Microsoft Fabric which is included as part of this course in the section 1.

All the very best and Happy Learning!!!

Who this course is for:

Are new to Azure Data Engineering and want to learn from scratch.
Data enthusiasts looking to gain practical experience in Azure Data Engineering.
Professionals seeking to expand their skills in data ingestion, transformation, and analysis.
Individuals interested in building data analytics platforms using Azure services.

Microsoft Fabric: End to End Data Engineering Project

What you'll learn

Explore related topics

Course content

Introduction (Pre-requisites)2 lectures • 29min

Build an End to End Project using Microsoft Fabric13 lectures • 3hr 13min

Requirements

Description

Who this course is for: