Advanced Retrieval Augmented Generation

Explore how tokens function as subword units, not full words, and how token IDs, input tokens, and output tokens drive LLM processing and OpenAI pricing.

Switch from synchronous to asynchronous code to run multiple OpenAI LLM calls concurrently, using async IO and gather to fire coroutines and await them, speeding up many calls.

Implement asynchronous disk caching for LLM calls by creating stable cache keys from function arguments, handling hits and misses, and storing JSON-serialized results to speed up repeated prompts.

Improve autocompletion for the cached chat completion by wrapping the underlying function with functools.update_wrapper, exposing its arguments (max tokens, temperature, etc.) and preserving the original signature for better IDE hints.

Learn to design robust AI workflows by using structured outputs that transform LLM text into programmable objects, enabling parsing, validation, and seamless passing between functions.

Learn to write and iteratively refine a prompt to generate a synthetic dataset for evaluation. Use a Jinja template to render prompts with placeholders and manage multi-line formatting.

Improve prompt design for retrieval augmented generation by classifying emails as useful or useless to recall and generating questions only from useful content, with batch iteration for faster experimentation.

Finish improving the prompt for evaluation question generation in retrieval augmented generation by refining it with ChatGPT, addressing attachment handling, and adding a second step to specify details for recall.

Execute robust OpenAI embedding calls using a text embedding model, implement caching to avoid repeated work, and prepare for cosine similarity calculations.

Compute embeddings for each question by iterating over the email dataset, building a twin data structure for back-and-forth navigation, and batch-generating embeddings with a cached embedding function.

Build a retrieval evaluation workflow using embeddings and cosine similarity to filter bad questions. Compare embeddings of bad questions with generated questions to flag deletions and review candidates.

Build a prompt schema to classify whether two questions are duplicates, using a duplicate verdict enum, a system and user prompt, and chain-of-thought reasoning to produce a final JSON verdict.

Execute the duplicate identification prompt to compare question pairs, filter duplicates, and generate deduplicated questions from the dataset, starting with a subset before full deployment.

This lecture demonstrates improving question quality in retrieval augmented generation by using LLM with structured outputs, evaluating standalone questions, and rewriting prompts based on email content and a JSON schema.

Download section 0205 resources (notebook, written questions json, poll csv) and attach to the first lesson; run all cells to verify questions and set stable keys via a simple namespace.

Install lensdb, the open-source vector database, and enable full-text search with bm25 via tenkV. Create a disk index for emails using Tantivy and update requirements and gitignore the lensdb directory.

Connect to a LanceDB database, create a table with email data, and build a full-text search index on the content column using mode overwrite, then verify with a quick search.

Measure retrieval performance by adding a reranking step to full text search, comparing reranked results to raw results, and evaluating Colbert and cross encoder models for top results.