Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Realtime Streaming End to End Azure Data Engineering Project
Rating: 4.7 out of 5(13 ratings)
73 students

Realtime Streaming End to End Azure Data Engineering Project

End-to-end real-time Azure streaming with Databricks, Event Hubs & Microsoft Fabric (Eventstream, Eventhouse, Power BI)
Created byMr. K Talks Tech
Last updated 10/2025
English
English [Auto],

What you'll learn

  • Design a real-time streaming architecture on Azure using Databricks, Event Hubs, and Fabric end-to-end
  • Ingest live Weather API data and publish events to Event Hubs using PySpark producers.
  • Implement an alternative ingestion path with Azure Functions and compare with Databricks.
  • Configure Fabric Eventstream to route events to Eventhouse (Kusto) reliably.
  • Model data and write KQL in Eventhouse for fast querying and aggregations.
  • Build live Power BI dashboards on Eventhouse with low latency and incremental refresh patterns.
  • Trigger real-time alerts using Data Activator based on streaming conditions.
  • Apply cost-management tactics and make architecture trade-offs for production-grade systems.

Course content

7 sections • 11 lectures • 5h 27m total length

  • Project Overview8:47

    Create a real-time weather reporting system ingesting data from weather API and streaming through Azure Databricks, Event Hub, Microsoft Fabric, and Kusto, with email alerts and a Power BI dashboard.

Requirements

  • An Azure Pay-as-you-go subscription
  • No prior Databricks/Event Hubs/Fabric experience required—taught in course.

Description

Build a production-style real-time streaming data pipeline on Azure, right from Data Ingestion to live dashboards. In this hands-on project, we ingest Live Weather API events through two paths (Databricks PySpark and Azure Functions), publish to Azure Event Hubs, process with Microsoft Fabric Eventstream, land data in Eventhouse (Kusto), and visualize insights in Power BI with real-time alerts via Data Activator.

You’ll learn how to provision the environment correctly (Resource Groups, Databricks workspace/cluster, Event Hubs namespace & hub, Key Vault, Fabric workspace), manage secrets securely, and wire services together for a reliable streaming architecture. We’ll write KQL for fast analysis, and apply cost-savvy choices for clusters, Throughput Units, and storage.

By the end, you will be able to:

  • Design a secure, scalable streaming architecture on Azure.

  • Implement ingestion with Databricks (PySpark) and Azure Functions.

  • Stream events into Event Hubs and route them with Fabric Eventstream.

  • Load/query data in Eventhouse using KQL.

  • Build live Power BI dashboards and trigger alerts with Data Activator.

  • Monitor, troubleshoot, and optimize for cost and performance.

  • Perform final end-to-end pipeline testing, from data ingestion all the way to the report updating with the latest data.

This project is perfect for anyone who wants a clear, practical understanding of how streaming ETL pipelines are built in real-world projects.

Who this course is for:

  • Data engineers who want an end-to-end real-time streaming Azure project.
  • Analytics/BI engineers building live dashboards on Fabric / Power BI.
  • Students and career switchers needing a portfolio-ready streaming build.

Report abuse