Ho to Test and Evaluate AI Agents - an introduction

Install Visual Studio Code for Java development and configure it with Maven, Debugger for Java, and GitHub Actions to support continuous testing and continuous integration.

Install python via the microsoft store to auto-resolve dependencies and avoid manual PATH changes; verify with python --version in terminal and use version 3.11 for benchmarking ai agents.

Install pip to manage python packages by downloading get-pip.py and running it on your system. Use pip install to add dependencies like openai and prepare for the hands-on demo.

Learn how to obtain and securely manage your OpenAI API key, create a new secret, and use it for all demos and ChatGPT requests.

Learn what AI agents are as orchestrators that coordinate prompts, language models, code executors, databases, and tools, and explore types like prompt chaining and workflow routing.

Explore the architecture of AI agents by examining six core components: interface, large language model, goal engine, action, tools, and memory, and how monitoring and observability enable testing and debugging.

Explore the non-deterministic testing of ai agents, shaped by context, memory, input-output links, and tool integrations, yielding varied outputs.

Analyze how the degree of AI agent autonomy drives testing approaches, from predetermined steps to highly autonomous tool creation and fact validation across domains.

Learn how to design and test prompts for large language models. Focus on system prompts, zero-shot, few-shot, and chain-of-thought strategies to set boundaries and improve results.

Explore few-shot and one-shot prompts, using given-when-then examples to tailor large language models, control tone, and output formats such as JSON.

Explore chain of thought prompting, comparing zero-shot and thinking-in-steps approaches, and learn when to apply tool selection like web search, rag, or databases for deep research and agent design.

Explore the MCP architecture, a transport layer that decouples tools from code and enables a host to discover and orchestrate server APIs. Test determinism and reliability in LLM tool use.

Explore the MCP architecture where one agent discovers tools on remote servers and uses the agent-to-agent protocol to orchestrate task handoffs across agents.

Explore evaluation approaches for AI agents, from rule-based automated testing to human evaluation and LLM-as-judge methods, and learn Ragas metrics for topic adherence and tool calls.

Learn about the goal completion metric, a boolean that indicates if the agent fulfilled the reservation goal (0 or 1), evaluated over multiple attempts using databases, prompts, and API bookings.

Learn how to measure goal completion as a boolean 0/1 metric for AI agents, illustrated by reservation booking across databases, rules, and API calls.

Break goals into tasks, and test each task in turn using available tools and multiple LLMs, then perform a system integration test to verify overall goal completion.

Learn how AI agents select the correct tool from a toolbox of MCP and NCP hosted services, ensuring tool call correctness through task-specific testing and integration checks.

Learn how to orchestrate tool selection and sequencing for AI agents, using a restaurant-booking example to show the order: rag database, rules engine, schedule/reservations, and final verification, plus cross-model variability.

Learn how to enforce instruction following and constraint adherence in AI agents, evaluate rules and ethics, and manage tool use like voice calls and reservation APIs.

Explore API contract compliance for AI agent integrations by generating and validating payloads, selecting the correct endpoints, and handling errors across restful, soap, grpc, and xml and json payloads.

Demonstrate plan creation correctness in multi-agent orchestration by simulating product search and order placement tools. Tests verify tool selection, parameter correctness, and inter-agent planning from search to payment.

Store per-user conversations to maintain context across sessions, since APIs have no memory, and test memory by using Microsoft Autogen caches to retrieve history.

Explore re-entry strategies to resume AI agents from the last stopped point and ensure idempotent actions yield consistent results after failures.

Learn how to implement stop conditions and loop control for AI agents, using max turns and termination checks to stop conversations after a correct answer or predefined iterations.

Analyze latency and time-to-result by measuring total execution time from start to result, including inference time, API calls, and database steps. Test components independently and monitor concurrency and token usage.

Explore token costs in AI agent deployments and learn to count prompts and responses with token counters for formats like JSON, XML, and YAML.

Explore vertical and horizontal scalability for ai agents, ensuring high availability and geo redundancy. Validate concurrency, parallelism, and session affinity with JMeter, Locust, and K6.

Explore how the MCP architecture enables extensibility and modularity by discovering, adding, and removing decoupled tools via the MCP protocol and server-side toolset, without retraining or fine-tuning.

Learn how retrieval augmented generation uses external knowledge via a retriever, augmentation, and generation to improve accuracy, extend memory, and protect private data.

Explore applications of retrieval-augmented generation (rag) by building chatbots that fetch order status or answer FAQs from vector and non-vector databases, using prompts, generators, retrievers, and a foundation model.

Explore how to validate retrieval with the ragas framework, comparing prompts, model responses, and references using metrics like context precision, context recall, and faithfulness.

Discover context precision, a rag metric that measures the proportion of relevant chunks in retrieved context using precision at k. The penicillin discovery example illustrates true positives and false positives.

Learn to test context precision with ragas evaluate by installing dependencies, building a ground truth dataset, running rag-cli tests, and interpreting answer relevancy and context precision averages.

Explore how context recall measures retrieved relevant information and avoids missing results, contrasting it with precision using the penicillin discovery example (who, when, where).

Demonstrate context recall in a rag-based pipeline using ragas library, showing retrieved context, references, and answers with print statements and threshold adjustments, plus the total context recall score.

Assess answer relevancy by comparing a response to the question using context precision, context recall, and cosine similarity, as staged by the Ragas library and an LLM judge.

Demonstrate testing answer relevance with ragas using a CLI, ground-truth references, and cosine similarity, while an LLM acts as judge to score and validate relevance.