Generative AI for QA Engineers Agents, RAG, LLM testing

In this lecture, you'll explore the critical role of routers in AI agent workflows—components that determine which tool, function, or skill the agent should invoke next. While often invisible, routing logic powers decision-making in almost all agent-based architectures.

What You’ll Learn:

• What Is a Router?
  Learn how routers guide the flow of decisions within an AI agent and why they're foundational to intelligent automation.

• How Routing Works:
  Understand the mechanisms behind routing—whether it’s through LLM function calls, intent classifiers, or rule-based logic.

• Evaluating Router Performance:

  • Was the correct tool, step, or function selected?

  • Were parameters accurately extracted and passed?

  • Did the agent route properly in complex or incomplete scenarios?

• Edge Case Handling:
  Test how routers behave when faced with missing inputs, ambiguous instructions, or multi-function triggers.

Lecture Description: Automation of Agent Plan Testing with Custom Metric

In this lecture, we explore how to automate the testing of AI agent plans using tailored evaluation metrics. You’ll learn how agent workflows are validated for correctness, consistency, and adaptability across dynamic scenarios. The session introduces key automation frameworks, test case generation methods, and the design of custom metrics to assess reasoning quality, decision accuracy, and tool usage. Through practical examples, we demonstrate how automated testing pipelines improve efficiency, reduce manual verification, and ensure reliable agent behavior at scale. By the end, you’ll understand how to build and integrate a custom evaluation layer that continuously measures and optimizes agent performance.

This lecture is designed for AI Engineers and Researchers who are tired of "vibes-based" evaluations and want to move toward a Scientific Evaluation framework for AI Agents.

We will explore how to quantify the reliability of complex agentic workflows, specifically focusing on how to evaluate a PlannerAgent's ability to maintain Plan-to-Goal Integrity (PGI).

Lecture Overview

As agents become more autonomous, traditional metrics like RAG-triad or simple accuracy fail to capture the nuance of multi-step reasoning. This session dives deep into the architecture of a Scientific Judge—an automated system that evaluates agent traces with the precision of a human auditor but at the scale of a machine.

Key Learning Objectives

• Deconstructing the Agent Trace: How to format Planner, Router, and Tool logs for optimal observability.

• Beyond Pass/Fail: Implementing PGI (Plan-to-Goal Integrity) and Execution Precision metrics.

• The Problem of Judge Stochasticity: Why LLM judges often provide inconsistent scores (the "Evaluation Variance" problem).

• Techniques for Reducing Variation: * Ensemble Judging: Using multiple LLM passes with $temperature = 0$.

  • CISC (Confidence-Informed Self-Consistency): Weighting scores based on the judge's self-assessed confidence levels.

  • Reasoning-First Rubrics: Forcing judges to extract evidence before assigning a numerical score.

In this lecture, you will learn how to systematically test and evaluate Retrieval-Augmented Generation (RAG) systems and Large Language Models (LLMs) using practical testing frameworks and metrics.

We’ll explore how to validate retrieval accuracy, context relevance, hallucination risks, and response faithfulness, ensuring that AI-generated outputs are reliable, grounded, and production-ready. You’ll also understand how to design automated evaluation pipelines, create custom test cases, and apply LLM-as-a-Judge techniques for scalable testing.

By the end of this lecture, you’ll be able to confidently assess the quality, correctness, and robustness of RAG and LLM-based applications in real-world scenarios.

Lecture: Mastering Claude Code – Commands, Context Management & Working Modes

In this hands-on lecture, you will learn how to effectively use Claude Code as an AI-powered engineering assistant by mastering its core commands, context management capabilities, memory mechanisms, and execution modes.

Claude Code is more than just a coding assistant—it is an agentic development environment that can understand project context, manage large codebases, execute workflows, and automate engineering tasks. To unlock its full potential, developers must understand how to control context, manage memory, and choose the appropriate operating mode for different tasks.

? What You'll Learn

Getting Started with Claude Code Commands

Learn the essential commands that help manage your Claude Code sessions efficiently:

• /init – Initialize a project and create project-specific instructions.

• /clear – Clear the current conversation context and start fresh.

• /compact – Reduce conversation size while preserving critical context to optimize token usage.

• Session management best practices for long-running engineering tasks.

Context Engineering in Claude Code

Discover how context directly impacts the quality of Claude's responses and agentic behavior.

You will learn how to:

• Add project-specific context effectively.

• Provide architecture, coding standards, and workflow instructions.

• Improve code generation quality through structured context engineering.

• Manage large codebases without overwhelming the context window.

Memory & Data Management

Understand how Claude Code stores and uses information at different levels.

Topics include:

• Project-Level Memory

  • Shared instructions for an entire repository.

  • Managing and updating CLAUDE.md.

• Local-Level Memory

  • Context specific to a machine or development environment.

  • Workspace-specific configurations.

• User-Level Memory

  • Personal preferences and reusable instructions across projects.

  • Standardizing workflows and coding conventions.

You will also learn:

• How to update stored instructions.

• When to clear data and refresh context.

• Best practices for maintaining clean and reliable memory structures.

Claude Code Operating Modes

Master the different execution modes and understand when to use each one.

Explore:

• Plan Mode – Research and planning without making changes.

• Default Mode – Human approval for actions and modifications.

• Accept Edits Mode – Automatic code edits with review control.

• Auto Mode – Autonomous execution with built-in safety checks.

• Advanced Permission Modes for automation and CI/CD environments.

Learn how to balance:

• Productivity

• Control

• Security

• Automation

for different engineering scenarios.

? Key Takeaway

The true power of Claude Code comes from effective context management, structured memory, and proper mode selection. By mastering these capabilities, developers can transform Claude Code from a simple coding assistant into a powerful AI engineering partner capable of supporting complex software development workflows.

By the end of this lecture, learners will be able to confidently manage Claude Code projects, optimize context and memory usage, and leverage the right operating modes to maximize productivity and code quality.