AI Chatbot Testing for QA : Build a Complete Test Strategy

In this lecture you will learn the complete CHART framework — five specific, testable, repeatable tests that every AI chatbot must pass before it reaches real users. Covering Conversational Flow Accuracy, Hallucination Detection, Accuracy of Classification, Response Range Validation, and Turn Retention — with real production examples for each.

In traditional QA, every test case has one expected result. AI chatbots do not work that way — the same question asked ten times will return ten different responses. In this lecture, you will learn how to write test cases that actually work for non-deterministic systems.

By the end of this lecture you will be able to:

- Replace the concept of "expected result" with acceptance criteria — a set of boundaries the response must stay within

- Write a structured AI test case with four components: required element, prohibited element, tone, and fallback behaviour

- Express your acceptance criteria as a Gherkin BDD scenario with consistency built in as the final condition

- Understand and apply confidence thresholds when your system exposes them

- Write a complete, production-ready AI test case from scratch — using the full worked example AI-TC-012

This lecture includes a downloadable AI Test Case Template worksheet so you can write your first AI test case immediately after watching.

Duration: 13 minutes 28 seconds

You have test cases. You have acceptance criteria. But without a test plan, every AI project will feel chaotic — and you will be re-explaining your approach to every stakeholder, every sprint, every time the model changes.

In this lecture, we walk through all seven sections of the AI Chatbot Test Plan Template — the document you downloaded in Lecture 1 — section by section. Have it open while you watch.

By the end of this lecture you will be able to:

- Write a Scope and Objectives section that protects you from out-of-scope blame — including the one sentence every AI test plan must contain

- Build a CHART Coverage Map — your traceability matrix proving all five behavioural dimensions are tested

- Organise your AI-TC test cases into a structured Test Case Library, grouped by CHART dimension for easy regression testing

- Write product-level Acceptance Criteria and exit criteria — the line between "we are done testing" and "we are not done yet"

- Document your Test Execution Approach so anyone on the team can run the tests

- Define four defect severity levels for AI-specific failures before the first defect is raised

- Complete the Ownership and Sign-Off section — the part most junior QA engineers skip and the one that protects you most

This lecture uses the AI Chatbot Test Plan Template (downloaded in Lecture 1). No new download needed.

Duration: 11 minutes 1 second

Finding a defect is only half the job. The other half is communicating it in a way that cannot be dismissed.

In this lecture, you will learn exactly how to run AI tests with discipline, write defect reports that hold up under scrutiny, and win the argument when a developer tells you a hallucination is "just model behaviour."

By the end of this lecture you will be able to:

- Apply the 3 rules of AI test execution that make your logs credible — 10 runs minimum, exact responses copied verbatim, logged immediately

- Write a complete AI defect report using all 7 required fields — the standard JIRA template does not work for AI failures, and this lecture shows you exactly what to use instead

- Assign defect severity correctly with written justification — not just a label, but the argument that explains why

- Know when to raise a Critical defect immediately without waiting for 10 runs — one confirmed hallucination is enough

- Win the developer pushback argument — when you hear "that's just model behaviour," here is the exact response that gets tickets accepted

- Use the product specification as your source of truth — not the developer's opinion about model limitations

This lecture includes the real story of a 40% hallucination rate, a developer pushback, and how a single line in the product spec resolved it.

Duration: 9 minutes 54 seconds

This is the final lecture — and it covers the maintenance challenge nobody warns you about when you start testing AI systems.

The AI changes. And nobody tells you.

No memo. No ticket. No Slack message. One day the model gets updated, a prompt gets quietly edited, or the knowledge base gets refreshed — and your tests start producing different results. This lecture gives you the framework to handle it.

By the end of this lecture you will be able to:

- Identify the three invisible changes that can silently break your AI tests — model update, prompt change, and data or RAG update — and understand why none of them trigger a release note

- Know which CHART dimensions to focus on for each change type — so you run targeted regression, not a full test cycle every time

- Set an AI regression schedule — monthly minimum, weekly for high-stakes systems, and always after any deployment

- Apply the living document mindset — version-control your test plan so every change is logged and traceable

This lecture closes with a full six-point recap of everything covered in the course and the one statement every AI QA engineer needs to internalise: an AI that passed your tests last month may fail them today.

Duration: 6 minutes 17 seconds