AI & Agentic Testing for QA Engineers

This course contains the use of artificial intelligence.

AI systems do not behave like traditional software. Their outputs can vary, retrieval can fail silently, agents can choose the wrong tool, and a passing functional check can still hide serious quality or safety risk. This course gives QA professionals a practical framework for testing those systems with confidence.

You will move from familiar software-testing concepts into modern AI evaluation. The course begins by translating assertions, test data, acceptance criteria, and regression testing into an eval-driven approach. You will then test LLM outputs for correctness, relevance, consistency, safety, and non-deterministic behavior using deterministic checks, model-graded evaluation, and human review.

Using a runnable QA Shop Assistant sample application, you will investigate real failure modes instead of relying only on theory. You will build evaluation datasets, validate Retrieval-Augmented Generation (RAG), measure retrieval quality, test faithfulness and grounding, and catch hallucinations. You will also test agent behavior, including tool selection, tool arguments, returned results, multi-step trajectories, and permission boundaries.

The security section covers the expanded attack surface of AI applications, prompt injection, red-team thinking, Promptfoo-based checks, and safety and policy evaluations. You will then bring these practices into delivery pipelines by running evals in GitHub Actions, defining thresholds and quality gates, identifying regressions, and planning production monitoring.

By the end of the course, you will be able to design a complete evaluation strategy for an AI-enabled product and assemble a practical test suite that combines functional quality, RAG validation, agent verification, security checks, and release criteria.

This course is designed for manual testers, QA engineers, SDETs, test leads, and QE architects who want to expand into AI and agentic-system quality engineering. Basic software-testing knowledge is helpful, but prior AI testing or LLM automation experience is not required. The downloadable lab package includes the sample application and setup instructions used throughout the practical exercises.