Kafka DBT Databricks AI (LLM) Integration SIX PROJECTS LIVE

Learn to dynamically import the housing prices dataset from kaggle hub into databricks, create raw, clean, and prediction tables, and prepare features for linear regression models using sklearn.

Write training code for random forest regressor using pandas, numpy, split data with train_test_split, define features and target, handle missing values, drop target, and evaluate with mse, mae, and r-squared.

Create predictions on the test split, attach actual and predicted prices to each row, and save the results to delta tables for easy validation and performance review.

Create a new housing dataset with demography, city, and weather features, then train the V1 housing prediction model on this data to evaluate performance.

Import numpy, pandas, and PySpark SQL functions, load the clean delta table, and enrich the dataset with city, season, crime rates, year build, and synthetic demographics.

Join city demographics into housing data, generate reproducible synthetic distance features to the CBD, schools, and hospitals, and save the enriched data as a delta table for model training.

Perform feature engineering on enriched data to derive age from date of birth, compute price per square foot, and create location quality index, conduct exploratory data analysis to examine dependencies.

Split the data into 80% training and 20% testing sets, apply a pipeline with multiple models, perform 5-fold cross-validation, and evaluate with rmse, mae, and r2 to select best pipeline.

Debug and evaluate models with cross-validation to select the best gradient boosting, generate predictions, publish to a delta table, and register and load the model from mlflow.

Build and configure a housing price prediction chatbot by loading a pre-trained model, defining input features and city enrichment data, and setting up configuration with mlflow and feature engineering.

Define input fields in a housing prediction chatbot by building utils to convert scalars to series, load the mlflow model, and prompt for strings, ints, and yes/no inputs.

Build a feature engineering function to enrich dataframes with season mapping from sale month, defaults, missing value handling, amenities index scores, price per square foot, and house age.

Develop a feature engineering function that creates beds-to-baths ratio, parking effect with log transform, school rating index, city enrichments, and a location quality index from amenities and CBD distance.

Align incoming data to the exact training schema with alignToTrainingSchemaDFBD.DataFrame, setting defaults for base columns. Ensure engineered features exist via the feature engineering pipeline and reorder to the training order.

Align incoming data to the exact model training schema by building a one-row feature data frame. Fill missing values and prepare features for price prediction with the trained model.

Debug pipeline errors and run a chatbot with structured queries to estimate prices, adjusting models, parameters, and prompts for cities like Karachi and Islamabad.

Learn how to generate and copy a Databricks Open AI token from the developer access tokens panel and use it in a notebook for a Databricks hosted open source LLM.

Explore how to normalize LLM outputs to meet chat bot v1 expectations by configuring a Databricks environment, loading an MLflow model, and wiring the OpenAI client with tokens and endpoints.

Develop and run a free-form text chatbot that extracts a compact JSON of real estate features from English descriptions using an LLM, then predict price with a model.

Explains how to integrate dbt with Databricks using dbt cloud, outlining core concepts, materializations, testing, documentation, and deployment through a browser-based studio and ci cd pipelines.

Learn to fetch cryptocurrency data from CoinMarketCaps API with Python, stream it into Red Panda Kafka, and configure access control lists (ACLs) and permissions for Databricks users.

Build a function to fetch coin market data using a Confluent Kafka setup, configure API requests, handle errors, and prepare data for publishing to Kafka — part 1 of 2.

Design and implement a Python function that fetches coin market data from coin market cap and streams it to Kafka as JSON messages, with error handling and payload flattening.

Fetch continuous coin market data and publish to a Kafka topic, including price, volume, market cap, and multi-period changes; read the stream in Databricks and clean with dbt.

Fetch data from Kafka with a PySpark consumer, define a Spark schema for incoming JSON, and batch-write 500 records every 60 seconds to a Databricks Delta table.

Transform delta table data with dbt to generate a data lineage diagram while debugging delta merge errors, configuring delta writes, and validating consumer-driven data flow in Databricks.