Complete RAG Testing course with Ragas Deepeval and Python

Install python via the Microsoft Store to automatically resolve dependencies, verify with python --version in both the Microsoft Store and the terminal, and avoid manual path edits.

Learn how to install the Python pip utility, download get-pip.py, run it to install pip, and use pip install to add dependencies like openai for hands-on demos.

Install node.js and npm from nodejs.org using the prebuilt installer, verify with node -v and npm -v, and ensure your path includes node and npm bin paths.

Create a new secret API key from the API keys panel, name it demo, and copy the key for all demos and ChatGPT interactions. Keep it private; do not share.

Understand the two main types: task-specific weak AI and general-purpose AI, with examples like computer vision and ChatGPT, and contrast them with artificial general intelligence.

Discover how ai introduces a new layer in a traditional decoupled app, integrating model APIs, orchestration, and a vector database for RAG.

Foundation models are versatile ai models trained on diverse data with a very good algorithm and transformer architecture, and large language models are the language-focused subset of these models.

Discover how retrieval augmented generation extends model memory with external data, reduces hallucinations, and uses vector databases and embeddings. Learn the three-part rag flow: retrieval, augmentation, and generation.

Explore how hallucinations arise in RAG systems and implement truthfulness testing to verify AI outputs against verifiable sources.

Explore four popular rag techniques—standard rag, corrective rag, speculative rag, and graph rag—plus knowledge graphs and multi-document k-top scoring.

Explore rag (retrieval augmented generation) applications, building chatbots that answer order queries and policy faqs by querying vector databases with embeddings and a retriever–generator pipeline.

Clone the rag test repository from GitHub, create a rag test folder, install dependencies from requirements.txt with pip, and prepare the environment (venv or conda) for a simple rag pipeline.

Implement a retriever with faiss using mupdf for vector embeddings and chunking, embed documents with an OpenAI embedded model, and store them in an on-disk vector database for interactive retrieval.

Learn how prompts are tokenized into tokens, converted into vector embeddings, and retrieved from documents by chunking with a defined size and overlap for effective top-k results.

Explore implementing an augmentor in a retrieval-augmented generation workflow by combining a system prompt with the top three retrieved contexts to guide a foundation model.

Explore how to retrieve top k documents or chunks from a knowledge base in a rag system, using a vector database and cosine similarity to fetch the most relevant documents.

Identify the most relevant document, retrieve its top three chunks, and use a two-stage rag approach with metadata and cosine similarity for accurate answers.

Explore how a vector database uses cosine similarity on token embeddings to retrieve top-k document chunks, and test retriever performance with precision and recall.

Explore augmentation testing for RAG by combining user prompts, retrieved top-k chunks, and a system prompt into a well-formatted, larger prompt for the model.

Test the generation stage by feeding prompts into a llm, assess precision, recall, and truthfulness, and verify contextual accuracy using the ragas library and deep eval.

Explore manual testing of rag pipelines, using human evaluation and a ground truth test oracle to probe questions like speed dial behavior on devices, highlighting slow, prototyping only limitations.

Learn automated API validations with Pytest to build fast, replicable tests that probe responses for keywords, while addressing non-deterministic AI and flaky results.

Explore specific query synthesizers in rag testing, using precise, single-hop retrieval to extract named entities from document chunks, with contrast to abstract queries.

Master multi-hop reasoning for complex, specific queries by navigating linked information chunks from the retriever and validating information across sources.

Explore multi-hop abstract query synthesizers for RAG testing, retrieving information across multiple document chunks and grounding abstract meanings in the source materials.

Explore golden nugget metrics in rag testing by examining how retrievers fetch chunks from a knowledge base, identify golden nuggets, and evaluate recall and support scores.

Learn how to construct the ground truth table as the reference for evaluating entity recognition, then compare predicted tables to compute F1 score and accuracy using a test oracle.

Define accuracy as the proportion of correct predictions over total predictions to gauge model performance. Beware imbalanced data, as small, unrepresentative tests can mislead due to garbage in, garbage out.

Recall equals true positives divided by true positives plus false negatives, showing how often positives are detected. It matters when missing positives is costly (medical diagnosis, fraud).

Learn how the F1 score harmonically averages precision and recall to evaluate models, especially with imbalanced classes, using a cancer patient example that highlights true positives, precision, and false negatives.

Explore the ragas validation framework for retrieval, learn to install ragas, and evaluate your retrieval-based llm applications with metrics like context precision, context recall, and bleu/rouge comparisons against a reference.

Learn to test context precision with the ragas evaluate library, building a ground-truth dataset and measuring answer relevancy to achieve a context precision score.

Demonstrates context recall in a python demo using ragas, showing reference, context, and answer, with a 0.5 threshold and a context recall average of 0.857.

Explore RAG metrics like context relevance, precision, recall, and relevancy, and learn how this library uses generated questions and cosine similarity of embeddings judged by an LLM.

Explore faithfulness as a RAG metric for truthfulness, linking hallucination risk to retrieved context. Identify claims, verify them against the reference context, and compute the context-supported ratio.

Explore faithfulness validation in RAG testing, examining how retriever results, contexts, and references influence truthfulness scores, and how chunk size and overlap affect results.

Explore the retrieval, augmentation, and generation stages of a rag system, validate retrieval with a vector database, integrate ChatGPT with context, and evaluate context recall and precision.

Evaluate rag pipelines using fluency, coherence, relevance, and concision, with an LLM as judge and deep eval benchmarks for reliable evaluation.

Implement an html report to visualize test results, exportable as csv or xml for ci/cd, and display per-test metrics like fluency, coherence, and relevance.