Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Kafka DBT Databricks AI (LLM) Integration SIX PROJECTS LIVE

Name: Kafka DBT Databricks AI (LLM) Integration SIX PROJECTS LIVE
Rating: 4.0 (7 reviews)

Learn HOW to Integrate KAFKA DBT DATABRICKS and Artificial Intelligence(LLM) by creating SIX projects LIVE

Created bySreenivasulu Kandakuru, M Umer Tariq

Last updated 1/2026

English

What you'll learn

Build Real-Time Data Pipelines with Kafka
Transform Analytics Data Using DBT
Process Streaming & Batch Data with Databricks
Integrate AI with LLM-Powered Chatbots

Course content

12 sections • 47 lectures • 9h 3m total length

Overview of Course14:26
Provides an overview of the 6 projects which will be coded live during this course

Code Walkthrough : Housing Prediction V214:35
Provides a code walkthrough of Housing Prediction Enhancement project
Add more features to Housing Prediction V1 model e.g demography , city , weather5:19
Create Dataset (Do NOT depend on testing team)14:05
Join the new features to existing dataset11:16
Feature engineering enriched data e.g derive age from dob column12:48
Exploratory Data Analysis using matplotlib and seaborn on newly created dataset9:30
Exploratory Data Analysis - continued9:06
Split Train and Test and KFold Framework .Run Check F1 scores17:05

CodeWalkthrough -> Chatbot V14:49
Provides a code walkthrough of chatbot v1 . uses structured queries to chatbot
Import Libraries and Configuration to use Housing Prediction Model14:24
Define input fields in chatbot on the Housing Prediction Model12:48
Build Feature Engineering function 1/214:30
Build Feature Engineering function 2/221:14
Align incoming data to exact schema used in Model Training 1/28:33
Align incoming data to exact schema used in Model Training 2/219:35
Debug issues and finally Run Chatbot with structure queries5:15

Code Walkthrough -> ChatbotV2 using LLM Prompt Engg and OpenAI5:42
Provides a code walkthrough of chatbot v2 which is AI enabled. Supports unstructured queries (e.g freetext like whatsapp) to chatbot using OpenAI , LLM.
Setup to access Databricks Open AI Token using Databricks hosted open source LLM2:19
Normalize output from LLM feature to what the Chatbot V1 expects (1/2)9:09
Normalize output from LLM feature to what the Chatbot V1 expects (2/2)12:30
RUN the chatbot using free form text (e.g whatsapp , facebook messenger)23:40

CodeWalkthrough5:41
Provides a code walkthrough of a project which pulls data from CoinMarket API and loads to a Kafka Topic and loaded to Databricks Delta lake.
6.1 Call to get Coin Market data and configuration in Databricks7:02
6.2 Build a function to hold Coin Market Data - 1/220:37
6.3 Build a function to hold Coin Market Data - 2/221:13
6.4 Continuous Stream Data from Coin Market API stored to a Kafka Topic14:48

CodeWalkthrough5:27
Provides a code walkthrough of the previous project by adding DBT to provide modularity in SQL. governance features such as table level lineage and column level lineage diagram is also shown which is nowadays critical
7.1 Fetch Data from Kafka and Store in Databricks Delta Tables 1/214:00
7.2 Fetch Data from Kafka and Store in Databricks Delta Tables 2/215:08
7.3 Fetch Data from Kafka and Store in Databricks Delta Tables in "Append mode"27:07
7.4 Delta Table data transformed using DBT to generate Data Lineage Diagram15:39

Requirements

Initial level of Python Programming experience with SQL required

Description

Used my experience and struggles while learning from online videos. There are lot of simple projects in the web whereas

in real work atmosphere you need depth and integration experience handling multiple DATA tools.

Hence thought of building upon projects with enhancements would the best way to CUT THE CLUTTER in becoming a TRUE Databricks Integration Expert. Someone who is not only expert in Databricks but knows how to integrate messages using kafka, build modular SQL using DB, ingest rest API and delivery AI capabilities (LLM) as well

SUMMARY

1. LIVE coding of End-to-End Data Engineering (Not Just Tools)

The core teaching is how data flows from source to insight, covering:

Event-driven ingestion with Apache Kafka
Analytics transformations with dbt Labs
Unified batch + streaming processing using Databricks

Learners understand why each component exists, not just how to click buttons.

This is engineering-first, not slideware.

2. AI-Augmented Data Engineering with LLMs

A major differentiator of this course is teaching:

How to build an LLM-powered chatbot connected to pipelines
Using AI to explain transformations, detect data issues, and assist debugging
Applying LLMs for self-service analytics and platform observability

AI is taught as a practical accelerator, not abstract theory

List of Projects which we will build as we talk through in the videos LIVE coding:

PROJECT 1 Start with a simple project . Housing Prediction V1 uses scikitlearn.

PROJECT 2 Enhance PROJECT 1 . Housing Prediction V2 uses multiple ML models and around 40 features.

PROJECT 3 Enhance the project further by adding chatbot using LLM with structured queries

PROJECT 4 Enhance PROJECT 2 even further by making it enterprise ready. Here we add chatbot using LLM with unstructured queries (free text like whatsapp) using LLM and OpenAI

PROJECT 5 Here we focus on integration skills integrating Streaming Messages with Databricks

PROJECT 6 Here we focus on integration skills integrating Streaming Messages with DBT and Databricks with enhancements

Become DATABRICKS Integration expert (NOT only databricks tool) quickly by doing DATA INTEGRATION PROJECTS

Who this course is for:

Data Engineers (Junior → Senior) Professionals who want hands-on, production-style experience building real-time and batch data pipelines using Apache Kafka, dbt Labs, and Databricks, with a strong focus on live coding and real-world patterns.
Analytics Engineers & BI Developers Practitioners responsible for transforming raw data into analytics-ready models, who want to: Move beyond static SQL Learn dbt best practices Understand how streaming data flows into analytics layers
Advanced Students & Career Switchers Learners who: Already know SQL and basic Python Want industry-grade, hands-on experience Are preparing for real-world data engineering roles
AI / ML Engineers & Data Scientists (Applied Focus) Learners interested in integrating LLM-powered chatbots with data platforms to: Query pipelines in natural language Explain transformations and data quality issues Build AI-assisted data engineering workflows
Not Ideal For Absolute beginners with no SQL or programming background Learners looking for theory-only or tool-specific certification cramming

Kafka DBT Databricks AI (LLM) Integration SIX PROJECTS LIVE

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 14min

OVERVIEW OF PROJECT 1 and PROJECT 21 lecture • 4min

PROJECT1 -> Housing Price Prediction V17 lectures • 1hr 1min

PROJECT2 -> Housing Price Prediction V28 lectures • 1hr 34min

OVERVIEW of PROJECT 3 and PROJECT 41 lecture • 6min

PROJECT3 -> Chatbot V18 lectures • 1hr 41min

PROJECT4 -> ChatbotV2 using AI5 lectures • 53min

OVERVIEW of PROJECT 5 and PROJECT 62 lectures • 24min

PROJECT5 -> Get Coin Data using Kafka and process using DBT2 - V15 lectures • 1hr 9min

PROJECT6 -> Get Coin Data using Kafka and process using DBT - V25 lectures • 1hr 17min

Requirements

Description

Who this course is for: