E2E Testing ChatBot, AI Agent, RAG, MCP Server with DeepEval

discover how the DeepEvalve platform provides a large language model evaluation framework with pytest-style unit testing, llm-as-judge evaluations, 50+ research baked matrices, and multi-model support, plus the ConfidentAI portal.

Install DeepEval via pip, set up the coding playground and environment with Python, VS Code, and your Windows, Mac, or Linux OS, and prepare OLAMA's local language model.

Set up the macOS environment for testing AI apps by configuring a Python virtual environment in Visual Studio Code, activate it, and install DeepEvalve via a requirements file.

Install and configure DeepEval on Windows 11 by setting up Python, Visual Studio Code, creating a virtual environment, resolving execution policy, and installing dependencies via a requirements file.

Write and run your first DeepEvalve test with exact-match evaluation, observe a passing result, and learn when LLMs act as judges versus non-llms like pattern match and JSON correctness.

Write and run a simple llms-as-a-judge based evaluation using DeepEval's answer relevancy matrix, creating a test and configuring an openai api key in a .env file.

Learn to use Ollama with a visual interface to run local large language models, compare models, upload documents, and generate code with a GUI without internet.

Explore the Ollama command line to manage and inspect models, remove and list them, and view details like architecture and context length, showing how it mirrors docker for local workflows.

Deploy Ollama as an API server on port 11434 to access a large language model via the generate endpoint.

Explore llms as a judge, evaluate local llms with olama, publish results in the confident ai portal, and extend olama with DeepEvolve library to disable thinking for faster non-thinking llms.

Explore the concept of large language model as a judge to evaluate other LLM-based applications, using predefined criteria and prompts in DeepEval to rate chatbot and AI agent outputs.

Learn to run app evaluation in sync mode by disabling asynchronous execution in Jupyter, resolve async errors, and configure a local large language model as the evaluation judge.

Evaluate llm applications using local llms instead of openai gpt by configuring the answer relevancy matrix with an olama model in deepeval and running tests locally.

Read environment variables from an env file in Python, replacing hard-coded values with local ollama model and local ollama base url using the os library.

Push local test results to the Confident AI portal by authenticating with an API key, logging into DeepEvalve, and viewing answer relevancy metrics for test cases in one place.

Extend the OllamaModel with no think to disable reasoning, speeding up local runs in DeepEval and simplifying the generate methods in Python.

Explore using actual large language model outputs for testing, and explain how to draw information from such applications with LangChain concepts for end to end testing.

Replace hardcoded outputs with actual llm.invoke results in a notebook, using the DeepSeq R1 model for both app output and evaluation, and track telemetry in the .deepavail folder.

Learn to run multiple llm test cases with the llm test case class in deepeval by passing a list of test cases to the evaluate method, enabling single-step testing.

Explore data sets for evaluating llm applications by combining llm test cases with goldens, enabling runtime evaluation across models and data-driven testing with open source data sets.

Create a simple evaluation data set in deep eval by adding test cases from csv, json, or from scratch. Generate goldens for future evaluation.

Create and manage goldens in a data set for test data in DeepEval, using the golden class with input, output, and context, then push and pull to the portal.

Create golden data sets from JSON inputs and expected outputs to run robust evaluations of LLM applications, using diverse data sources like Kaggle, Hugging Face, and Adversal QA benchmarks.

Create a goldens dataset for adversarial QA with 6000+ records by extracting context, questions, and answers from json, then push to the Adversal Network Dataset for evaluation.

Learn to provide full retrieval context to a large language model by crafting LangChain system and human messages. Manage test data and run evaluation with DeepEval for RAG chatbots.

Learn to evaluate a large language model with DeepEval using built in matrices and custom gevalve metrics. Explore dag matrices, criteria, scoring, and test organization in a metrics folder.

Extend the GEval bias testing workflow by creating a bias matrix and custom evaluation steps in DeepEval, and assess gender bias in model outputs.

Explore contextual precision metrics in DeepEval, learn to define and test them with rag pipelines. See how retrieval context and expected versus actual outputs validate model performance.

Introducing testing of a next-gen ai app with Next.js front end and FastAPI back end, using DeepEval, PyTest, a LangChain pipeline, and a local OLAMA deployment.

Explore the smart card assistant, an AI-powered chatbot that handles product queries, cart updates, and comparisons using chroma db and a local gamma model, tested via deep eval.

Understand the application tech stack for an e2e testing chatbot, including next.js frontend, fast API backend, sqlite data, chroma db vector store, and nomic embed text v2.

Learn to test an ai application end-to-end by validating api calls with postman. Explore normal and similarity searches, embeddings, chroma db, and the rag pipeline behind the ui.

Discover how to build and test an AI-driven application backend using a Postman API collection for products, carts, and orders, including semantic search and chat APIs powered by llms.

Test the chat API powering ai applications with deep eval, validating chat bot responses and chroma db data. Explore semantic and similarity search for product retrieval and add-to-cart interactions.

Set up and run the e2e testing chat bot locally by installing Python, npm, Visual Studio Code, then start frontend and backend using gamma 482 billion parameter model and nomic-embed-text-v2-moe.

Explore testing your application with DeepEval basics, run large language models locally, and use embedding and RAG with vector stores and fast API-backed services.

Design end-to-end tests for AI apps by testing chat endpoints with unique session IDs, posting messages, and validating streaming responses via server-sent events.

Define a get_cart method that calls the get cart endpoint with a session id using requests.get and a 10-second timeout, returning the json of cart items with product details.

Write tests to verify the chatbot and cart flow using a new session, deep eval, and custom GEVAL, validating chat responses and cart confirmation end-to-end.

Explore testing an app with deep eval, covering local LLM deployment, embedding and search with RAG, API interactions, and moving tests from notebook to pytest.

Move boilerplate code from a deep eval notebook to pytest conftest and test files, organize tests into folders, create reusable fixtures, and streamline the chatbot test workflow.

Explore using conftest.py and pytest fixtures to share fixtures across classes and modules, automatically invoked by pytest, and assemble a test_add_card flow with chat, get_card, and judge.

Learn to fix test failures and harness pytest conftest fixtures to share session id and judge across tests, enabling reusable, configurable test code with scoped fixtures.

Explore pytest fixture scopes to reuse objects across tests, such as a judge and session id. Learn how session scope can optimize multi-test executions in a single class or file.

Explore fixture scopes and grouping tests in PyTest by implementing a Python test class, writing test methods like test_card_count, and using session scope to illustrate cross-test impact on cart items.

Configure pytest to log in to the Confluent AI portal using an environment variable API key, then push DeepEval test results for portal reporting.

Leverage pytest parameterization to validate multiple chat intents for cart operations, such as clear, empty, and remove, by applying arrange, act, and assert steps and reviewing the test dashboard.

Explore updating cart quantities via an AI chat bot, surface a deliberate bug, and compare model reliability using Olama cloud and the DeepSeek V4 Pro for end-to-end testing.

Parameterize an update cart test to evaluate a chat bot, adding items like trial running shoes, ddr4 ram, and wireless airbird, then update quantities and verify the cart.