AI Agents, RAG & LLM Evals for Beginners: DeepEval & RAGAS

Learn to select and run local models with Ollama, from 7 billion to 671 billion parameters, examining quantization, headcount, and hardware requirements on a capable machine.

Learn to run llm locally with Ollama: list and pull models, download a 1.8b parameter Alibaba Cloud model, and switch to the deep sea 8b model for offline code generation.

Start Ollama as an api server and interact with a llama model via http endpoints on port 11434, using api generate calls and optional streaming to retrieve model responses.

Set up the environment to test large language models with deep eval and ragas; create a Python virtual environment, install deep eval, and enable Jupyter in VS Code.

Learn to set up and use a Jupyter notebook in Visual Studio Code. Select the Python kernel, install the ipy kernel, run code, and save notebooks for DeepEval testing.

Learn to set up a confident ai login for the deep eval cloud and enable evaluation of large language models with a data set, simulations, observability, evaluation, and human feedbacks.

Compare large language model testing to traditional software testing, and apply unit, functional, regression, and responsibility testing using deep eval to assess accuracy, bias, toxicity, and fairness.

Use large language models as judges to evaluate LLM-based applications, such as chatbots and AI agents, with metrics like answer relevancy, faithfulness, and bias detection.

Write the first deep eval code to test an llm with the answer relevance metric, loading the OpenAI key via env or dot env, using Joe Biden as context.

Explore deep eval's context precision matrix and answer relevancy metrics to evaluate LLM outputs, using test cases with input, actual output, and retrieval context.

Run live evaluations with deep eval using the evaluate method and push results to the Confident AI portal; view insights, LLM test cases, inputs, outputs, and costs on the dashboard.

Evaluate multiple test cases with the evaluate method to measure answer relevancy metrics using a retrieval context, and perform an A-to-B comparison of LM outputs.

Explains that a data set in DeepEval stores goldens, the truth used to generate test cases, and shows converting goldens to test cases for evaluation with an evaluation data set.

Create a golden dataset, build a data set with test cases from it, include actual output, and evaluate using input, expected output, and retrieval context with relevancy metrics.

Formulate and manage golden datasets from multiple test cases using the Confident AI portal. Learn to push, store, and reuse evaluation datasets with inputs, outputs, and context.

Pull and manage a cloud data set from confident AI to generate golden test cases, convert them to LM test cases, and evaluate with a mock LMS app.

Learn to run evaluations with a local large language model using Ollama and DeepEval, configure the local model, and use the local model as a judge for assessment.

Learn to run evaluations using local LLMs with the deep eval tool, switching from GPT to the deep eval r1 8b Olama model and managing runs for testing.

Learn to build a golden data set by using a local large language model for evaluation and actual output generation, integrating LangChain to run tests with a local LLM.

Learn to use LangChain to invoke a local LLM with Ollama, configure a notebook workflow, install LangChain and llama libraries, and generate evaluation data using invoke operations.

Evaluate a local large language model (LLM) as both application and judge to test answer relevancy, using lang chain, LM invoke, and prompt engineering to compare outputs and retrieval context.

Evaluate local large language models with contextual precision metrics and threshold control in a real Ollama setup. Observe retrieval context handling and bias types to validate outputs against expectations.

Evaluate bias in local LLMs by applying a bias matrix with a tunable threshold, running bias metrics, and checking prompts for gender bias.

Use the gavel framework in deep eval to create custom evaluation metrics, with an LLM as judge and chain of thought, to assess bias and faithfulness.

Build and evaluate custom bias metrics with GEval in the GA framework for local LLMs, using DeepEval tests and AI agent safety metrics.

Explore how retrieval augmented generation (rag) improves large language model accuracy by retrieving external data before answering, and learn to test rag-based apps with deep eval.

Describe how a RAG-based app extracts data from PDFs and other sources, embeds it into a vector store, and uses retrieval augmented generation, with testing by DeepEval.

Demonstrates testing a rag application that reads from vector stores, chunks data with embeddings, and uses the llama 3.2 latest model with the model context protocol to answer questions.

Create a GEval test case for a rag application using DeepEval, building an evaluation data set and an LM test case to ensure concise and complete outputs match expected results.

Explore testing a RAG app with GEval to measure completeness and conciseness using DeepEval, custom and answer relevancy matrices, and an evaluation dashboard.

Advance testing of rag applications with deep eval, using multi-input/output and golden data sets, while managing actual and expected outputs in confident AI in VS Code.

Create and manage multiple test data for evaluation by building input-output data arrays, using golden data sets, and preparing a pipeline to push results to confident AI for future lectures.

Create golden data sets and push them into Confident AI, overwriting existing datasets, run evaluation data sets, and prepare retrieval context and outputs from rag and MCP components.

Pull the golden data into test cases, populate actual outputs and retrieval context using a Rag workflow, retriever, and retrieval QA with LangChain to build LM test cases.

Create lm test cases from a golden data set by converting each golden input into a test case with input, actual output, expected output, and retrieval context.

Evaluate test cases against deep eval matrices using answer relevance, faithfulness, context precision, and contextual relevance metrics to verify RAG performance, compare retrieval contexts with expected outputs, and assess results.

Understand how ai agents use the model context protocol to access data and tools, enabling tool calls and browser automation. Learn to test ai agent interactions with deep eval.

Demonstrates testing an AI agent by binding tools like the add tool, the subtraction tool, and the DuckDuckGo search tool to produce real-time responses.

test an ai agent using deep eval by creating an lm test case, sending queries to a 2.5 model, and validating tool calls, inputs, and outputs.

Test an ai agent with deep eval using a local llm model, set up tool calls and test data, and verify the correct add numbers tool invocation to yield 60.

Create tool correction metrics and use the measure method to verify test cases offline. Note the deep eval limitation: online evaluation isn't available, so results come from offline testing.

Test AI agents with the actual system instead of a fake system by using the query AI agent method to obtain real tool inputs, outputs, and responses.

Demonstrate testing ai agents with multiple datasets and tools, including the DuckDuckGo search, to verify correct tool invocation. Measure data-driven evaluation with DeepEval and track tool calls across test cases.

Move DeepEval testing from Jupyter notebooks to the PyTest framework to reduce boilerplate in AI agent and RAG evaluation.

Migrate code from jupyter notebooks to the pytest framework and set up a virtual environment. Create a requirements.txt with langchain, langchain-core, and deepEval 4.0.5.

Learn to run local LLMs for evaluation without embedding a local model in tests by centralizing settings in a dot env file and resetting DeepEval metrics configurations.

Leverage Conf Test in PyTest to remove boilerplate from DeepEval test code by moving setup into a dedicated fixture file, simplifying tests and wiring automatic environment loading.

Explore pytest hooks and fixtures in conf test to automatically login to the DeepEval llm app, and move boilerplate setup into reusable configurations for llm-based testing.

Learn to organize PyTest tests into Python classes to improve readability and scalability for DeepEval tests, including class structure, self parameter, and class-based test execution.

Learn how to use Python closures to reduce boilerplate in DeepEval tests, migrate from Jupyter to pytest, and run faster RAG evaluations with LangChain and smaller models.