Databricks Certified Generative AI Engineer Associate

Short presentation of the chapter.

Databricks Free Edition

A short onboarding introduction to the Databricks Free Edition, launched in June 2025.

Connect GitHub to Databricks

This hands-on connects a GitHub repository to the Databricks workspace in three steps — generating a GitHub personal access token with repo scope, linking it to Databricks, and importing the repository as a Git folder to enable full version control from the workspace.

Prompt

This presentation covers how to interact with an LLM through prompts — from the basics of the OpenAI message format, prompt templates, and the six prompt types, through to advanced techniques like few-shot in-context learning, chain-of-thought reasoning, and context pruning.

LLM — Foundational Models :

An educational video covering foundational LLMs in the Databricks ecosystem, aimed at practitioners new to the space.

Model lifecycle :

A visual walkthrough of the end-to-end AI model lifecycle on Databricks, structured as a progressive build.

MLflow on Databricks :

A presentation designed to sell MLflow to a technical audience, opening with a reframe — MLflow is not "yet another platform to learn" but a unified lifecycle manager for the AI agent era.

This session describes how to create the API key for the Claude model used in the hands-on.

Create & Manage LLM Credentials

This hands-on securely stores LLM API keys in Databricks using the SDK — creating a secret scope, adding API keys that can be referenced in code without ever exposing the actual value, and verifying the setup confirms that Databricks automatically redacts secret values in all outputs and logs by design.

Create a Claude Serving Endpoint

This hands-on creates a production-ready external model endpoint for Claude Sonnet 4.5 using the MLflow Deployments SDK — configuring the endpoint name, deploying it programmatically before testing it in the AI Playground.

Simple LLM Chain with MLflow Tracing

This hands-on sends four Databricks-related questions to the "claude_45" endpoint via a @mlflow.trace-decorated wrapper function that automatically records inputs, system prompt, response, and execution time as structured traces — then retrieves and inspects those traces both inline in the notebook output and in the MLflow UI, demonstrating the trace-first observability pattern that underpins all subsequent chain and RAG development.

Short presentation of the chapter.

Governance Introduction

This presentation makes the case for building governance into agentic AI systems from day one — covering the four failure modes of ungoverned AI (security risks, compliance gaps, observability blind spots, and lost business value) and defining Agent Governance as a lifecycle-spanning layer of control, visibility, security, and risk management.

Unity Catalog

This presentation introduces Unity Catalog as Databricks' centralized governance layer — walking through its three-level hierarchy (catalog → schema → objects), the objects it manages (tables, volumes, views, functions, models), fine-grained access controls including row-level filtering, column masking and ABAC, and a secure-by-default permission model based on standard ANSI SQL privilege inheritance.

Create PDF Volume

This hands-on sets up the Unity Catalog data infrastructure for the RAG application — creating a governed  Volume, and copying our knowledge-base PDFs (product catalog, return policy, shipping guide, technical FAQ) from the project GitHub repository, making them immediately available for the RAG pipeline.

Create CSV Feature Tables

This hands-on loads  e-commerce CSV files (customers, orders, order items, products, shipping zones) from GitHub into a Unity Catalog Volume, then transforms each into a production-ready Unity Catalog table, with schema, description, and lineage metadata visible in the Data Explorer.

Agent Identity Governance

This presentation explains why agents must have their own dedicated identity — a Databricks service principal — rather than borrowing human credentials, and shows how service principals provide minimal-permission access, clean audit separation, job execution independence, and scalable group-based identity management aligned to agent roles.

AI Gateway

This presentation covers the Unity AI Gateway as the production control plane for all model endpoints — providing rate limits, cost tracking, A/B traffic splitting, automatic fallbacks, inference table logging, and two-way guardrails (Meta Llama-backed safety filtering and Microsoft Presidio-powered PII detection) to keep agents secure, observable, and compliant at scale.

Marketplace, NOAA Analysis & Genie

This hands-on covers the full journey from data discovery to insight in three steps:

1. browsing the Databricks Marketplace to find the Rearc NOAA Daily Weather Observations dataset, reviewing its US federal copyright licence and non-redistribution terms of use, and installing it as a Delta Sharing catalog in Unity Catalog with one click;

2. then loading the shared table via a single Unity Catalog path reference, applying transformations  and producing a monthly bar chart comparing precipitation days across selected cities for 2023;

3. before finally reproducing the same insight in seconds by querying Databricks Genie in plain English, demonstrating that the same analysis is accessible to non-coders through natural language SQL generation.

Genie Natural Langage Query :

`for the year 2023, represent in a bar chart the number of days per month where precipitation exceeded 50 (i.e. 5 mm, since values are stored in tenths of mm) for the following stations: USW00023234 (San-Francisco), USW00012916 (New-Orleans), and USW00013743 (Washington). Group results by station and month, and display city names instead of station IDs.`

Short presentation of the chapter.

In this lecture, I will introduce a new stage and its role in a RAG chain, the "Retrieval".

RAG Chain

This presentation introduces RAG as the solution to the monolithic LLM's inability to answer from unseen data — walking through each component of the complete chain: an optional query-understanding step, a retriever that fetches relevant context from a knowledge base, a context-enrichment step that augments the prompt, the LLM that generates the response, and an optional post-processing component that transforms the output before returning it to the user.

RAG Workflow & Unstructured Data Pipeline

This presentation maps the full RAG workflow — showing how an unstructured data pipeline (corpus ingestion, parsing, metadata enrichment, deduplication, chunking, embedding, and indexing) feeds the RAG chain through two linking components — the embedding model and the vector store index — while noting that multimodality, evaluation, and governance complete the production picture.

Chunking

This presentation explains why chunking — splitting documents into smaller text units — is a critical RAG design decision, then walks through strategies from fixed-size splitting with overlap and recursive character splitting, to advanced semantic chunking (grouping sentences by meaning) and LLM-powered approaches like IBM Docling and the Databricks `ai_parse_document` function, each offering different trade-offs between simplicity, retrieval quality, and compute cost.

Prepare RAG Data

This hands-on demonstrates two PDF chunking strategies for building the vector store foundation of a RAG system — first applying LangChain's Recursive split stategy from four e-commerce PDFs, then applying IBM Docling's chunker to produce semantically richer, structure-aware chunks — saving each strategy's output as a separate Delta table, before recommending the following trade-off: recursive for speed on simple docs, Docling for tables and complex layouts.

AI Parse Document

This hands-on first loads a diverse set of multiformat sample files (a 1908 French newspaper PDF, a 1976 insurance invoice JPG, an accident statement PDF with nested tables and handwritten fields) from GitHub into a Unity Catalog volume; it then parses each file using the ai_parse_document SQL function before rendering the results through a bounding-box overlay that visually audits element detection.

Embeddings

This presentation explains how embedding models convert text chunks into high-dimensional vectors that capture semantic meaning, then covers the two proximity search algorithms used to find the nearest vectors: the exact but linearly-scaling KNN, and the approximate HNSW (Hierarchical Navigable Small World) algorithm — which Databricks uses in Mosaic AI Vector Search — offering logarithmic search complexity by traversing a multi-layer proximity graph built at index time.

Vector Store

This presentation explains the distinction between a vector index (the HNSW-based search structure) and a vector database (the full data management system with CRUD, metadata filtering, governance, and horizontal scaling), then details Mosaic AI Vector Search's three index types — Delta Sync with Databricks-managed embeddings (auto-sync, auto-embedding), Delta Sync with self-managed embeddings (you supply pre-computed vectors), and Direct Vector Access (manual sync via REST API for maximum cost control) — across two endpoint tiers: standard (320M vectors, automated scaling) and storage-optimized (1B+ vectors, 10–20x faster indexing at 250ms latency).

Retriever

This presentation details how a modern RAG retriever combines keyword search (TF-IDF and BM25) with semantic search (embedding-based ANN), merges both ranked lists via Reciprocal Rank Fusion with a tunable beta weight, applies metadata filtering for governance and recency, and optionally applies an LLM-based reranker — which Databricks has integrated with a single parameter that improves recall@10 from 74% to 89% at a cost of only ~1.5 seconds of added latency.

Create Vector Search Endpoint

This hands-on creates the full Mosaic AI Vector Search infrastructure — provisioning a STANDARD endpoint via the VectorSearchClient SDK based on a Delta Sync index that automatically triggers a Delta Live Tables pipeline to embed each document chunk and store the resulting vectors in a vector column.

RAG Chain Wrapper

This hands-on decripts a production-ready RAG chain — enabling MLflow automatic tracing, loading all parameters from a YAML config file, assembling the full chain using LangChain Expression Language.

RAG Log & Load

This hands-on packages the RAG chain for production by logging it to MLflow then reloading the logged model from its URI to validate correct execution, inspecting the MLflow trace, verifying the correct PDF source appears in retrieved document metadata, and outlining the path from logged model to Unity Catalog Model Registry to Model Serving endpoint.

Short presentation of the chapter.

Agent

This presentation defines what makes a system "agentic" — the LLM's autonomy to choose tools and plan sequences of actions — then walks through the five architectural components of every agent (perception, reasoning core, orchestration, memory, and tools), illustrates them with a step-by-step financial consultant example, introduces the multi-agent supervisor pattern for complex enterprise tasks, and closes with real-world customer deployments including 7-Eleven, Fox Sports, Adidas, and the US Navy.

Agent Build

This presentation covers the three ways to build an agent on Databricks — AI Playground (no-code prototyping), the Mosaic AI Agent Framework (code-first with LangChain, LangGraph, or OpenAI SDK, integrated with MLflow and Unity Catalog), and Agent Bricks (automated, domain-specific agent builder that auto-optimizes chunking, embeddings, retrieval, and LLM selection) — and shows how existing agents can be migrated to the platform without rewriting by wrapping them in the MLflow ResponsesAgent interface.

Agent Tools

This presentation covers the full landscape of agent tools on Databricks — Unity Catalog SQL functions for predictable structured queries, Genie multi-agent systems for dynamic natural-language SQL, vector search retrieval tools for unstructured data, and the Model Context Protocol (MCP) as the emerging universal standard — distinguishing between managed MCP servers (pre-configured access to Genie, Vector Search, and UC Functions), external MCP servers (third-party services like GitHub or Slack secured via Unity Catalog OAuth proxies), and custom MCP servers hosted as Databricks Apps.

Agent Tools

This hands-on creates and registers two types of agent tools :

1. SQL functions that queries the e-commerce table, automatically registered via CREATE OR REPLACE FUNCTION

2. and a Python function, registered via DatabricksFunctionClient.

3. verifying both in the Unity Catalog Data Explorer,

4. and validating tool-calling behaviour interactively in the AI Playground.

Genie Agent

This hands-on builds and optimises a  natural-language assistant :

1. first creating the Genie space connected to five Unity Catalog business tables, testing it with a simple customer lookup.

2. then enriching metadata by AI-generating column descriptions for each table and adding conifguration instructions; then testing the enriched agent with dedicated queries.

Natural langage queries :

`Can you provide details for customers with the last name 'Smith'?`

`What is the distribution of the amounts of orders ?`

`Can you provide details for customers with the last name 'Smith'?`

Text instruction :

When users ask about customers, always refuse to include the registration_date in the response for privacy policy reasons.

Query instruction :

NL

`What is the distribution of the amounts of orders ?`

SQL

SELECT

  COUNT(*)                       AS order_count,

  MIN(`total_amount`)            AS min_amount,

  MAX(`total_amount`)            AS max_amount,

  ROUND(AVG(`total_amount`), 2)  AS avg_amount,

  ROUND(STDDEV(`total_amount`), 2) AS std_amount

FROM

  `demo`.`demo`.`orders_feature`

WHERE

  `total_amount` IS NOT NULL

Connect GitHub to Databricks with Managed OAuth

In this hands-on, we set up the GitHub MCP server on Databricks using Managed OAuth — no personal access token to create, no OAuth app to register.

Databricks handles all authentication flows and token refresh for you through Unity Catalog.

What you'll learn

• How to create a shared connector with OAuth User to Machine Per User auth

• How to test GitHub tools live in the AI Playground

Elements used in this video

• Comment:

"A list of tools available to interact with GitHub through an MCP server"

• Playground test questions

"List all the repositories in my GitHub account and tell me which one was updated most recently."

"For my user and the databricks_genai_demo repo, can you count the commits per week over the last month?"

Agent Wrapper

This hands-on walks through the LangGraph agent Python script — explaining the architecture :

1. LangGraph StateGraph for conversation flow,

2. Unity Catalog Function Toolkit for tools,

3. Vector Search retriever for unstructured knowledge,

4. the AgentState  that manages message history and optional custom inputs/outputs.

Agent Evaluation

This hands-on runs the complete agent evaluation.

1. locally testing the LangGraph agent with MLflow autolog, logging the agent as a versioned MLflow artifact with all dependencies.

2. then loading the artifact and wrapping it in a predict function;

3. building an evaluation dataset from a GitHub CSV;

4. running mlflow.genai.evaluate() with four scorers (RelevanceToQuery, Correctness, Safety, and a custom "is written" Guideline) and diagnosing failures

5. improving the agent by injecting a structured step-by-step system prompt via YAML config update, re-running evaluation,

6. and finally registering the validated agent in Unity Catalog via the MLflow UI for governance-controlled deployment.

Databricks supervisor,

In this Databricks Hands-on, you will discover the Databricks Supervisor Agent: a multi-agent orchestrator that coordinates specialized sub-agents to solve complex tasks.

In the first part, we will introduce the agent bricks UI. But the main part will use Mosaic AI to build a supervisor orchestrating a genie agent and a Unity Catalog function

Short presentation of the chapter.

Why & How to Evaluate

This presentation establishes why latency and throughput metrics alone are insufficient for AI systems and introduces the three evaluation approaches — aggregate stats, detailed trace logs powered by MLflow Tracing, and controlled experiments — before drawing the essential distinction between offline evaluation (testing before deployment against a labeled dataset) and production monitoring (running the same scorers on live traffic to catch quality drift continuously).

Evaluation Components

This presentation walks through the four infrastructure pieces Databricks provides for evaluation — MLflow experiment tracking (logging every run's parameters and metrics for comparison), MLflow Tracing (step-level visibility into every request from dev through production, optionally persisted to Delta via the Inference Table), the metrics framework (quality, cost, and latency per component using hosted LLM judges), and the Agent Evaluation Review App (a no-code interface for subject-matter experts to provide thumbs-up/down feedback or label structured evaluation datasets).

Evaluation Dataset

This presentation explains that an evaluation dataset — a curated collection of inputs and optional ground-truth expectations (expected facts, expected response, guidelines, and expected retrieved context) — is the gold standard that must exist before the first POC, then covers how to build one via three paths: extracting real traces with `search_traces`, running a labeling session with domain experts via the Review App, or auto-generating a synthetic baseline from documents using `generate_evals_df`, with Databricks recommending starting at 30 questions, scaling to 100–200, and treating the dataset as a living artifact stored as a Delta table in Unity Catalog.

Evaluation Metrics

This presentation explains why traditional code-based metrics fail for open-ended LLM outputs and introduces the unified Scorer abstraction in MLflow 3 — the single interface covering all three evaluator types: code-based scorers (deterministic, fast, no LLM cost), LLM-as-judge scorers (semantic, flexible, no ground truth needed for most metrics), and human feedback (highest quality, lowest scale) — mapped across two quality dimensions (response quality: Safety, Correctness, RelevanceToQuery, RetrievalGroundedness; and component quality: RetrievalRelevance, RetrievalSufficiency) and applicable identically in both development evaluation and production monitoring.

Built-in LLM-as-Judge Metrics

This presentation covers Databricks' research-backed out-of-the-box judges available in MLflow 3 — two retrieval metrics (RetrievalRelevance checking per-chunk pertinence, and RetrievalSufficiency checking whether retrieved context collectively answers the query) and four response metrics (Correctness against ground truth, RelevanceToQuery detecting on-topic responses, RetrievalGroundedness as the primary hallucination defence, and Safety filtering harmful outputs) — plus the flexible Guidelines judge for custom pass/fail criteria in plain language, all callable via a single `mlflow.genai.evaluate()` invocation.

Custom LLM-as-Judge Metrics

This presentation explains how to define domain-specific evaluation criteria using `make_judge` — covering the four template variables (inputs, outputs, expectations, trace), three feedback value types (Literal categories, boolean, and Literal with numeric values), and three judge patterns (issue resolution using inputs/outputs, expected-behaviors checklists using expectations, and trace-based tool call validation) — and closes with the SIMBA-powered alignment workflow that automatically rewrites judge prompts to better match human expert ratings collected via the Review App.

Code-Based Scorers

This presentation covers the four patterns for deterministic, Python-based evaluation logic in MLflow — heuristic scorers (one-liner pass/fail rules decorated with `@scorer`), trace-level scorers (accessing span timing and token counts from the full execution `Trace` object), expectations-based scorers (keyword or fact presence checks against ground-truth labels from the evaluation dataset), and class-based configurable scorers (Pydantic `Scorer` subclasses with tunable thresholds per agent type) — all plugging identically into `mlflow.genai.evaluate()` and production monitoring.

Cost vs Quality

This presentation frames the cost/quality trade-off as a three-stage journey (prototype → scale → optimize) and covers four cost reduction strategies — model selection (smaller or fine-tuned models), token optimization (shorter RAG chunks, concise output instructions, prompt caching), provider benchmarking via leaderboards like BFCL, and dedicated GPU clusters at high volume — before mapping the latency spectrum from speed-first (e-commerce, <500ms) to quality-first (medical, <5% hallucination rate) use cases, with `mlflow.langchain.autolog()` providing the per-span cost and latency data needed to make every decision with evidence rather than intuition.

RAG evaluation

This hands-on loads the latest MLflow run from the experiment, navigates the trace hierarchy to extract RETRIEVER span metadata used by the custom source correctness scorer, then constructs an evaluation dataset. It then assembles the full evaluation pipeline with built-in judges, custom Guideline judges, and the source correctness scorer, runs mlflow.genai.evaluate() and analyses failures in the MLflow UI, before improving the chain with a richer system prompt, adding two new security judges, and closing with a side-by-side delta table comparing v1 and v2 scores.

Short presentation of the chapter.

From MLOps to LLMOps :

This presentation begins with the MLOps practice then extends to LLMOps. 

Because of the use of large language models, new challenges are addressed like prompt versioning, context management, inference cost optimization, RAG pipelines, fine-tuning workflows, and safety/alignment monitoring at scale.

Batch Deployment

This presentation maps the three deployment paradigms (batch at hours/days latency, streaming at seconds/minutes, and real-time at milliseconds) and makes the case that batch is always the right default when the data can wait, then details the two Databricks paths to batch inference: AI Functions — serverless, SQL-native, with over 10x speed improvement, covering `ai_query()`, `ai_classify()`, `ai_summarize()`, `ai_translate()`, `ai_mask()`, and `ai_parse_document()` usable in Lakeflow Pipelines, Workflows, and Structured Streaming — and Spark DataFrame UDFs for custom or deep-learning models registered in Unity Catalog via MLflow's `spark_udf()` pattern.

Real-Time Deployment

This presentation introduces Mosaic AI Model Serving as the unified REST API layer for deploying any model — custom MLflow-packaged models, Databricks-hosted foundation models (pay-per-token or provisioned throughput), and external models (OpenAI, Anthropic, Google) — with under 50ms overhead latency, serverless auto-scaling to zero at idle, blue/green zero-downtime version updates, and inference tables that automatically log every request and response to a Unity Catalog Delta table for quality monitoring, production debugging, A/B testing, and fine-tuning dataset creation, all centrally governed through the AI Gateway.

Monitoring on Databricks

This presentation explains how to maintain model quality in production by combining two complementary tools: inference tables (a built-in feature that automatically logs every request, response, timestamp, and metadata from a serving endpoint into a Unity Catalog Delta table for debugging, quality monitoring, and training data generation) and Lakehouse Monitoring (which attaches a data-profiling monitor to those inference tables to track statistical drift, performance metrics like accuracy or R², and distributional changes over time against an optional baseline, generating automated dashboards and configurable SQL alerts).

Lakeflow Jobs

This presentation introduces Lakeflow Jobs as the procedural orchestration tool of the Databricks Data Intelligence Platform (formerly Databricks Workflows), covering its three core concepts — the Task (a unit of work: notebook, Python script, SQL query, ML training, DBT project, or an SDP pipeline), the Job (the DAG container with if/else branching and for-each looping), and the Trigger (time-based scheduling or event-based firing on file arrival or upstream completion) — and its four monitoring layers: the Job Monitoring UI for real-time run status, the run detail view with DAG and Timeline for parallelism diagnostics, configurable Notifications via email, Slack, or Teams with SLA duration thresholds, and System Tables for SQL-queryable historical analysis and dashboard creation.

Lakeflow Spark Declarative Pipelines

This presentation introduces Lakeflow Spark Declarative Pipelines (SDP, formerly Delta Live Tables) as a declarative framework that reduces thousands of lines of Spark and Structured Streaming code to a few lines of SQL or Python, covering its five core concepts — Append Flows (exactly-once streaming ingestion with checkpoints), AUTO CDC Flows (SCD Type 1 and Type 2 change data capture with automatic out-of-order event handling), Materialized Views (batch flows with incremental refresh that recompute only changed partitions), Streaming Tables (Unity Catalog managed targets for streaming flows), and Sinks (external Delta or Kafka targets the pipeline does not own) — all orchestrated automatically within a Pipeline DAG governed by Unity Catalog.

Dashboard & Alert

The first part builds a live AI/BI Dashboard from scratch — creating two SQL datasets (one counting orders grouped by status, and the second summing amounts by date), then building visualizations to represent each dataset, before publishing the dashboard.

The second part creates a Databricks SQL Alert that enforces the business rule that only five valid statuses are allowed. If an unexpected status is detected, a notification is triggered to a chosen email address.

Dashboard queries :

"Order by status" dataset

SELECT

  status,

  COUNT(*) AS order_count

FROM demo.demo.orders_feature

GROUP BY status

ORDER BY order_count DESC

"Daily Sales Summary with Order Counts" dataset

SELECT

  order_date,

  SUM(total_amount) AS daily_revenue,

  COUNT(*) AS daily_orders

FROM demo.demo.orders_feature

GROUP BY order_date

ORDER BY order_date ASC

Alert query :

"Alert Invalid Order Status"

SELECT

  COUNT(*) AS invalid_status_count

FROM demo.demo.orders_feature

WHERE status NOT IN ('delivered', 'shipped', 'pending', 'processing', 'cancelled')

SDP — Generate Source Orders

This hands-on walks through the setup notebook that acts as a data generator — creating the Unity Catalog Volume "raw orders" that Auto Loader will monitor, then writing a batch of  order into it (including an order with an intentionally invalid "awaiting" status designed to trigger the SQL Alert downstream), and confirming the file landed correctly by browsing the Volume in the Catalog tab.

SDP — Descript

The first part explains the Python script powering the order ingestion pipeline — covering the `import pipelines as DP` entry point, the Bronze Streaming Table using Auto Loader, the Silver Streaming Table that chains from Bronze and handles null order dates, the sink that registers the external `orders_feature` Delta table the pipeline does not own, the Append Flow that wires Silver records into the sink, and the Gold Materialized View that aggregates order counts by status.

In this second part, we will run the Spark Declarative Pipeline from the editor — pasting the pipeline script, observing the DAG update in real time, previewing the four generated tables (Bronze, Silver, Materialized View, and Sink), and resolving the sink's first-run "omitted" status by triggering a full refresh.

Finally, we will verifie the end-to-end quality monitoring loop — running the "Alert Invalid Order Status" to confirm it flips to TRIGGERED when "awaiting" status is detected, checking the email notification received from Databricks, then navigating the DAG to interpret the entire pipeline.

Create Service Principal & Grant Privileges

This hands-on creates a dedicated service principal in Identity and Access settings, then programmatically granting permissions on the models and corresponding resources.

Service Principal — Deploy Agent

This hands-on deploys the LangGraph agent as a production serving endpoint under the identity of its dedicated service principal.

The AI Playground is then used to check the deployment validation.

In this video, we deploy a Streamlit chat app powered by Claude on Databricks, using Databricks Asset Bundles to manage the infrastructure as code. We walk through the full workflow — from cloning the repository and configuring the CLI, to deploying the app, verifying the service principal, and monitoring requests through the Unity Catalog inference table.
You can find a detailled description of the hands-on in the readme of the project.

Course conclusion.