How AI Tests Software

Become an agentic AI test engineer: find bugs, verify outcomes, and assess release evidence

Created byJason Arbon

Last updated 10/2026

English

What you'll learn

Distinguish artifact checks, interactive tests, and coordinated testing harnesses.
Provide source and business context to guide an AI agent toward relevant software risks.
Create reusable test intent and verify outcomes using recorded execution evidence.
Evaluate AI findings with skeptical review, personas, and evidence-based judgments.
Use testing evidence and remaining gaps to build a release recommendation.

Course content

6 sections • 74 lectures • 5h 54m total length

1.01 Welcome to How AI Tests Software3:41
1.02 Set up your AI coding agent4:25
1.03 Meet TestBucks. Find your first bugs.1:49
1.04 Checks, tests, harnesses, and confidence5:35
1.05 AI checks: start with the evidence4:53
1.06 Write a bug report that helps fix the bug4:55
1.07 Find issues in logs and recorded evidence6:06
1.08 Connect findings to your existing tools2:19
1.09 Give your agent useful context5:15
1.10 Many pages. Fewer duplicate reports.6:46
1.11 Ask focused questions2:36
1.12 When the assistant refuses5:02
1.13 Security and privacy checks1:58
1.14 Usability and accessibility4:14
1.15 Performance, visual quality, and experience3:34
1.16 Functionality, reliability, and compatibility5:41
1.17 Content, localization, data, and APIs4:46
1.18 Review AI-generated code2:52
1.19 Standards and compliance checks4:36
1.20 Reduce noise. Test state and relevance.3:21
1.21 Recovery, specifications, and parallel checks2:59
1.22 Part One: the quality-checking loop2:30

2.01 Interactive tests: act and verify3:10
2.02 Let the agent drive the browser4:29
2.03 Connect your existing testing tools3:37
2.04 Use frameworks and save results to your tools2:57
2.05 Adaptive tests and AI-first bug tracking3:39
2.06 Why generated test code still breaks3:40
2.07 Save the intent, not the brittle path2:43
2.08 Executable intents and the OpenTest.ai schema4:37
2.09 Preserve evidence and explain the result3:22
2.10 Turn evidence into a quality decision2:55
2.11 Part Two recap: interactive AI testing0:37

3.01 Give your agent source and business context5:06
3.02 Challenge the implementation and its boundaries3:41
3.03 Prioritize risks with source-aware analysis3:48
3.04 Formal verification and Lean: where AI helps3:25
3.05 Check a focused property and its counterexamples4:07
3.06 The changing role of the tester3:03
3.07 People and AI: review complementary findings4:17
3.08 Generate test data and combinations at scale3:41
3.09 Manage review overload with sampling4:45
3.10 Balance model cost, capability, and speed6:00
3.11 Run local models with Ollama1:18
3.12 Local models still have biases and risks2:14
3.13 Fine-tune a model on your triage examples3:14
3.14 Compare cost, parallelism, and availability2:51
3.15 Personas and skeptical review3:35
3.16 Challenge your assumptions about AI testing3:05
3.17 Different paths, stable outcomes3:37
3.18 Check meaning across varying responses2:15
3.19 Temperature, top-k, and top-p3:04
3.20 Judge meaning with evidence2:40
3.21 Compare and calibrate independent judges1:24
3.22 Judge against evidence, not authority3:11
3.23 Part Three recap: contextual testing1:23

Requirements

Basic familiarity with web applications, software defects, and an AI coding agent such as Claude, Codex, ChatGPT, or Cursor.
For hands-on practice, a computer with a browser, an AI coding agent, and an environment you are authorized to test. Tool costs and availability vary.

Description

Become an agentic AI test engineer by learning how to use AI agents to investigate software, exercise user journeys, and evaluate the evidence behind a finding. In How AI Tests Software, Jason Arbon walks through practical testing workflows using examples such as the fictional TestBucks application.

Start with the distinction between checks, interactive tests, harnesses, and confidence. Learn to inspect screenshots, logs, and other artifacts, give an agent useful context, reduce duplicate findings, and write bug reports that help someone investigate a problem. The course examines several quality concerns, including functionality, accessibility, usability, security, performance, and compatibility.

Next, explore agents that operate applications and verify outcomes. Lessons cover connecting existing testing tools, preserving executable intent, keeping evidence, and turning results into a quality decision. Contextual testing adds source code, business context, boundary analysis, test data, personas, skeptical review, model tradeoffs, and evidence-based judging.

The later parts examine reusable intent across surfaces, recovery and load-testing boundaries, coordinated testing workflows, parallel workers, and release recommendations. The final part explores changing experiences, proof-oriented testing, game testing, and software that tests and repairs itself.

This course is for testers, developers, and QA leaders who want to apply agentic AI to software testing. Bring basic familiarity with software defects and web applications. For hands-on work, use an AI coding agent and a practice environment you are authorized to test; tool availability and costs vary. You will be asked to examine evidence and limitations rather than treat an AI-generated answer as proof.

This course contains a promotion.

Who this course is for:

Software testers and QA engineers learning to use AI agents in their testing workflow.
Test automation engineers exploring reusable test intent and evidence-based verification.
Software engineers who want to investigate risks and verify behavior with AI assistance.
QA and engineering leaders evaluating evidence, workflow coordination, and release confidence.

How AI Tests Software

What you'll learn

Explore related topics

Course content

Part 1: AI checks — evidence and findings22 lectures • 1hr 30min

Part 2: Interactive AI tests11 lectures • 36min

Part 3: AI contextual testing23 lectures • 1hr 16min

Part 4: Tests become intent7 lectures • 34min

Part 5: The AI test harness6 lectures • 1hr 3min

Part 6: Changing experiences and future testing5 lectures • 56min

Requirements

Description

Who this course is for: