AI-Ready Data Engineering: Pipelines for RAG & Agents

Learn to tune four embedding model dials—language, scale and budget, privacy, hosting—using a decision tree and a storage estimator to pick a model that fits your data and costs.

Measure freshness across ingestion, embedding, and index lags to prevent stale answers in RAG systems. Align updates with freshness using p95 thresholds and refresh for hot, warm, and cold sources.

Learn a zero-downtime schema evolution strategy for vector indexes with blue-green swaps, additive changes, and explicit model version tagging to safely migrate embeddings.

Build a contract-validated ingestion job with a gate and watermark-driven pull, embedding valid docs to the vector store and routing bad docs to a dead letter table.

Explore how a green pipeline hid stale policy data due to a lying timestamp, and how content hash reconciliation plus guardrails restore freshness for RAG and agents.

Detect the scanned-document trap by flagging pages with near-zero extractable text and applying OCR at 300 dpi before embedding, using OCR confidence to protect retrieval.

Build a unified parser that ingests PDFs, HTML, and scans, auto-detects formats, routes to the right extractor, and outputs a single parsed doc as JSONL with provenance for gating.

Normalize gently to preserve meaning: apply NFKC, de-hyphenate, collapse whitespace, and strip boilerplate, while not lowering case, not killing punctuation, and preserving numbers and IDs; recompute the content hash.

Chunk data into retrievable, coherent clips with source metadata to optimize recall and embedding quality. Avoid too-long or too-short chunks to hit the Goldilocks sweet spot and prevent garbage embeddings.

Explore four chunking strategies—fixed, recursive, semantic, and structural—highlighting their rules, best use cases, and how to match strategy to document shape for coherent, scalable processing.

Balance chunk size and overlap in data retrieval by tuning two knobs, measure recall and cost, and apply token-based sizing with 256–1024 tokens and 0–30% overlap.

Attach a metadata envelope to each chunk to enable access aware retrieval, filtering, citations, and security per the governance data contract, using parent-child chunking for precise search and context.

Dissect how a finance assistant's miscalculation arose from a chunk boundary cutting a formula, and present a three-move fix: structural chunking, 15% overlap, and CI coherence tests.

Transform notebooks into a production pipeline for embeddings by implementing an incremental, idempotent, cost-aware, batched, deduplicated, and observable ETL flow of extract, chunk, embed, load with a state table.

Learn how idempotency ensures safe retries by using deterministic chunk IDs hashed from doc id, position, and content, with absurd insert-or-replace semantics and per-batch checkpoints.

Learn to perform re-embedding on a model change, using a blue-green zero-downtime swap with versioned embeddings and resumable backfills.

Shrink embedding costs by using a content hash cache and dedupe across documents, caching by model version and text hash, and batching with async backfills.

Reconcile counts at every stage to detect silent drops, debug three killers—duplicate chunk IDs, swallowed batch exceptions, and null chunks—and fix with robust hashing, loud error checks, and guarded embedding.

Explore how a vector store replaces brute-force search by storing embeddings with metadata and doc ids, building a fast index, and delivering nearest-neighbor results with approximate search.

Map the vector database landscape into bolt-on vs purpose-built options, comparing PG vector, Snowflake, and Databricks with Pinecone and Qdrant, highlighting scale, latency, and governance tradeoffs.

Use a five-question decision tree to select the right vector store for your data pipeline, balancing vectors, data gravity, latency, budget, and ops appetite.

Compare bm25 and vector retrieval in a hybrid pipeline, highlighting idf-based term scoring, length penalties, and rank-based fusion using reciprocal rank fusion (rrf) to merge lexical and dense results.

Combine multiple ranked lists using reciprocal rank fusion (rrf) by summing 1/(k + rank) contributions, with k commonly 60, so top positions earn more points across bm25 and vector lists.

Explore the retrieval quality stack from lexical BM25 to vector AN, RRF fusion, metadata and entitlement filtering, and a cross encoder re-ranker, delivering the top-5 citable context.

Understand the semantic gap between sql execution and business meaning, and fix wrong answers by aligning metric, grain, filter, and join definitions; use aggregate before you join and with clauses.

A retailer's self-serve dashboards reveal a 2.3 million gap between gross and net revenue caused by the model treating revenue as gross; the fix enforces a governed net revenue definition.

Seed a DuckDB schema with orders and refunds, build a naive text-to-sql loop to compute q3 revenue, then diagnose a red-mismatch failure caused by missing refunds rule.

Governance comes next by defining meaning to trust text-to-SQL conversions. Spot traps like undefined metrics, wrong grain, and join fan-out that cause miscounts.

A semantic layer acts as the translator, encoding your company's slang to define net revenue and active users once, enabling a single source of truth across tools.

Master how metrics and dimensions drive dashboards, using net revenue to define a complete metric with base grain, expression, aggregation, filters, and synonyms, avoiding anti-patterns.

Explore how semantic models become code in YAML, versioned in git with a pull request, diff, and a CI pipeline, defining metrics and dimensions for the DBT semantic layer.

Define a semantic model to ground net revenue from gross and refunds, map synonyms like revenue and top line, and constrain metrics and dimensions for accurate reconciliation.