What you'll learn

Develop understanding of confluent platform, be able to create Cluster, Topics, Managed Source & Sink connectors
Develop hands on skills on Azure Function App, Azure Functions
Able to create Cosmos DB Account, container, database and query cosmos db
Able to setup Cosmos DB & Power BI Integration, Azure Function Sink connector and Azure Function Integration
Able to setup Power BI to fetch data from cosmos and create a visual from data
Develop understanding of creating & deploying Azure functions using Visual Studio Code

Course content

7 sections • 12 lectures • 1h 32m total length

Introduction2:55
Build a near real-time streaming data pipeline using Confluent Kafka on Azure, connecting Datagen source to Cosmos DB via Azure Function Sync Connector, and visualize analytics with Power BI.

Confluent: Create Cluster, Topic, API Key4:23
Build a real-time data streaming pipeline by creating a Confluent cluster on Azure, generating an API key, and creating a Kafka topic named orders to host data.
Confluent: Create Datagen Source Connector6:00
Create a datagen source connector in Confluent to push order data into a Confluent topic, configuring API keys, JSON format, and the order schema for real-time streaming.

Azure: Create Function App using Visual Studio Code4:23
Create an Azure function app in Visual Studio Code using the Azure extension, selecting legacy consumption plan, West US two, and Python 3.12, then configure storage and application insights.
Azure: Create & Deploy Function using Visual Studio Code7:36
Create and deploy an Azure function in the function app using Visual Studio Code with an HTTP trigger, test it, deploy to Azure, and monitor logs, keys, and invocations.
Azure: Update Function to log & parse Incoming Confluent Kafka Message8:14
Update an Azure function in Python to log with logging.info and parse incoming Confluent Kafka messages. From the message, extract the nested value and prepare for Cosmos DB insertion.

Create Cosmos DB Account, Database & Container6:26
Create an Azure Cosmos DB account and orders container with a NoSQL database and partition key id. Push real-time data from Confluent Kafka to Cosmos DB using an Azure Function.
Update Function Code to insert records into Cosmos DB22:35
Update the Azure function to insert records into Cosmos DB using Python, enabling end-to-end real-time data flow from Confluent to Cosmos DB for Power BI analytics.

Review End to End Data Flow - Confluent to Power BI7:07
Verify the end-to-end data pipeline from the data source connector through Confluent topic, Azure function sync connector, Cosmos DB, to Power BI, and validate with logs and queries.
Wrap up & next steps3:35
Review the end-to-end near real-time data pipeline built with Confluent Cloud, from data source connectors to Azure Functions and Cosmos DB, with Power BI for analytics.

Requirements

Be able to create Azure Account for hands on practice & follow along
Be able to create Confluent Cloud Account for hands on practice & follow along
Be able to install Power BI for hands on practice & follow along
Be able to install Visual Studio Code for hands on practice & follow along

Description

In this hands-on course, participants will follow along step-by-step to build a real-time streaming data pipeline that sends data from Confluent Kafka using Azure Functions Sink connector to Azure Functions to Azure Cosmos DB and finally creating operational report using Power BI. This course is designed to provide practical, real-world skills by walking through each component of the architecture, ensuring that learners not only understand the concepts but also apply them directly in a cloud environment.

On the Azure Cloud Platform side, participants will gain experience working with several important services, including Azure Functions, Cosmos No SQL DB and Power BI. The participant will also be exposed to logging, trouble shooting, monitoring and configuring these Azure Cloud Platform services. These services are essential for building secure, scalable, and reliable applications in the cloud.

On the Confluent Kafka side, learners will set up a fully managed Kafka cluster, create topics for message streaming, & configure a fully managed Source Connector to simulate real-world data ingestion, also configure fully managed Azure Functions Sink connector. This gives participants valuable exposure to enterprise-grade Kafka infrastructure without the overhead of managing the platform themselves.

By the end of this course, participants will have built a working, scalable pipeline, gained insights into cloud-native architectures, and acquired hands-on experience that can be directly applied to real-world projects or professional roles.

This course is suitable for budding Cloud Engineers, mid level Data Engineers, Product Owners, Product Mangers, Scrum Masters and Technology Leaders looking to get a hands-on experience of building a Real time Streaming Data Pipeline.

Who this course is for:

Entry Level Cloud developers looking to build skills in Confluent Kafka, Azure & Power BI space
Mid to Sr. Level Software Engineers looking to build data engineering skills by building Real time streaming data pipeline
Product Managers looking to get hands-on experience with Real Time streaming data pipeline using Azure + Confluent + Power BI
Technology Leaders looking to get hands-on experience with Real Time streaming data pipeline using Azure + Confluent + Power BI & gain appreciation for effort involved

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 3min

Confluent Setup2 lectures • 10min

Azure setup3 lectures • 20min

Confluent/Azure Integration1 lecture • 8min

Azure Cosmos setup2 lectures • 29min

Power BI1 lecture • 11min

Recap & Conclusion2 lectures • 11min

Requirements

Description

Who this course is for: