Testing AI Systems with DeepEval: AI Agents, Chatbots & RAG

Name: Testing AI Systems with DeepEval: AI Agents, Chatbots & RAG
Rating: 4.5 (1789 reviews)

Test AI Agents, Chatbots & RAG Apps with DeepEval using Metrics, Tracing, Goldens, Safety Evals & G-Eval Custom Metrics

Bestseller

Highest Rated

Created byRahul Shetty Academy - 1.2 Million QA Learners

Last updated 6/2026

English

What you'll learn

Understand the fundamentals of AI Testing and how it differs from traditional software testing approaches.
Set up and use the DeepEval framework to evaluate AI Agents, Chatbots, and RAG (Retrieval-Augmented Generation) applications.
Measure AI system quality using industry-standard evaluation metrics such as Answer Relevancy, Faithfulness, Precision, Recall, and G-Eval.
Create and manage Golden Datasets to build reliable and repeatable AI evaluation pipelines.
Perform trace-based and component-level testing to identify issues within AI agent workflows and reasoning chains.
Build custom AI evaluation metrics tailored to real-world business and product requirements.
Test multi-turn chatbot conversations and validate contextual understanding across complex interactions.
Evaluate RAG applications by assessing retrieval quality, context utilization, response relevance, and factual correctness.
Generate synthetic test data to improve AI evaluation coverage and reduce manual effort.
Perform AI Safety Testing to detect harmful, biased, unsafe, or undesirable model outputs.
Learn and apply Python for AI testing automation, with Python fundamentals covered as part of the course for beginners.

Course content

11 sections • 50 lectures • 7h 44m total length

Welcome to the Course - What this course is about?3:17
What exactly is Testing AI? How different it is from Traditional testing4:29
How to validate non Deterministic output AI Systems - Shift in testing mindset5:20
Introduction to DeepEval and its core capabilities for evaluating AI Systems10:51
Quiz - Test your knowledge

DeepEvals TaskCompletion Metric to evaluate the AI Agent - Blackbox Testing16:05
Understand reading the result reports from Confident AI - Deep evals Integration11:32
Important Notes1:08
Agents Component Testing overview - Understand internal workflow12:52
Goldens - Data sets to evaluate AI Systems and Importance to evals_Iterator14:30
Implement Tracing with in Test file for smart Agent calling & track workflow10:51

How to validate Multi Turn AI conversations such as Chatbot etc? - Overview5:53
Build Turns list object to track conversations and evaluate the Chatbot Metrics14:42
End to end demo of Multi conversational Agents validation with Deep evals5:43
Explore other Standard Deep evals Metrics to test Chatbot similar AI Apps7:43
Build Custom Metrics to validate Chatbot AI Systems using GEval - Example demo9:21
Quiz - Test your knowledge

What are RAG Agents? How different they are from Traditional AI Agents?16:31
Get demo of RAG App used for testing & compare it with AI Agent demo app11:28
RAG Metrics - Contextual Precision Metric to validate RAG Agent output quality12:46
Demo example of validating RAG Agents with standard Deep eval Metric methods15:42
RAG Metrics - Contextual Recall Metric to validate RAG Agent retrieval quality13:09
Quiz - Test your knowledge

Python hello world Program with Basics8:35
Datatypes in python and how to get the Type at run time5:17
List Datatype and its operations to manipulate12:47
Explore Python data types, including numeric types, strings, and lists. Learn list operations like indexing, slicing, insert, append, update, and delete, with dynamic typing and no explicit type declarations.
Tuple and Dictionary Data types in Python with examples8:28
If else condition in python with working examples3:10
How to Create Dictionaries at run time and add data into it7:55
How loops work in Python and importance of code idendation8:58
Programming examples using for loop - 14:17
Programming examples using While loop - 210:27
What are functions? How to use them in Python10:46
OOPS Principles : Classes and objects in Python7:38
What is Constructor and its role in Object oriented programming13:38
Understand how Python constructors are auto-called on object creation, including default and parameterized forms, and distinguish instance versus class variables with the self keyword.
Inheritance concepts with examples in Python12:12
Strings and its functions in python9:53

Requirements

Python basics are required. But wait! we have a dedicated sections at the end of this course which gives you necessary knowledge on Python & Pytest required to follow the course.

Description

Course Remade in June 2026 with latest Deep Evals Framework -- ~~Artificial Intelligence is rapidly transforming software applications, and traditional testing approaches are no longer enough to validate the quality of AI-powered systems. Whether you are working with AI Agents, Chatbots, Copilots, or Retrieval-Augmented Generation (RAG) applications, ensuring accuracy, reliability, safety, and performance has become a critical skill for modern QA and Engineering teams.

In this course, you will learn how to systematically test and evaluate AI systems using DeepEval, one of the most powerful open-source frameworks designed specifically for AI evaluation. Starting from the fundamentals, you will build a strong understanding of AI testing concepts and gradually progress toward implementing real-world evaluation strategies used in production AI applications.

Throughout the course, you will learn how to measure the quality of AI-generated responses using industry-standard evaluation metrics, create and manage Golden Datasets, perform trace-based analysis, validate AI agent workflows, and build custom evaluation metrics tailored to your business requirements. You will also explore G-Eval, component-level testing, multi-turn chatbot evaluations, and advanced techniques for assessing conversational AI systems.

The course further dives into testing Retrieval-Augmented Generation (RAG) applications by evaluating retrieval quality, response relevance, context utilization, and factual correctness. In addition, you will learn how to generate synthetic test data, automate evaluation workflows, and perform AI safety testing to identify harmful, biased, or unsafe outputs before they impact users.

By the end of this course, you will be able to design and implement comprehensive AI testing frameworks for AI Agents, Chatbots, and RAG applications using DeepEval. Whether you are a QA Engineer, Automation Tester, SDET, AI Engineer, Developer, or technology enthusiast, this course will equip you with practical, hands-on skills that are increasingly in demand as organizations adopt AI-powered solutions at scale.

Join me on this journey and learn how modern AI systems are tested, evaluated, and validated in the real world.

Who this course is for:

Software Engineers
Quality Assurance Engineers
Software Testers

Testing AI Systems with DeepEval: AI Agents, Chatbots & RAG

What you'll learn

Explore related topics

Course content

AI Testing Fundamentals: Thinking Beyond Traditional QA4 lectures • 24min

DeepEval Setup & Building Your First AI Testing Project4 lectures • 30min

Evaluating AI Agents with DeepEval Metrics & Golden Datasets6 lectures • 1hr 7min

Testing AI Agent Internals: Traces, Components & Root Cause Analysis3 lectures • 42min

⚙️ Building Custom AI Evaluation Metrics with G-Eval3 lectures • 30min

Testing Multi-Turn Chatbots & Conversational AI Systems5 lectures • 43min

Testing RAG Applications: Retrieval Quality & Response Accuracy5 lectures • 1hr 10min

Synthetic Data Generation & AI Safety Testing5 lectures • 33min

Final Exam - Assess your knowledge0

Optional - Learn Python Fundamentals with examples14 lectures • 2hr 4min

Requirements

Description

Who this course is for: