ISTQB AI Testing: Complete Training and Exam Preparation

AI-based systems have unique quality attributes that differentiate them from traditional software. In this lecture, we introduce key AI-specific characteristics, including autonomy, adaptability, non-determinism, and probabilistic behavior, and their impact on AI testing.

Unlike traditional systems, AI models learn and adapt over time. This lecture explains how AI flexibility impacts test planning, maintenance, and long-term validation. Learn how to test AI systems that evolve based on new data.

AI systems can make independent decisions and evolve without human intervention. This lecture explores the challenges of testing autonomous AI and how testers can ensure reliability and control over self-learning systems.

AI models can inherit and amplify biases present in data. This lecture introduces bias in AI systems, its causes, and how it affects fairness and ethical decision-making in machine learning applications.

In this video, I break down two specific types of bias that can impact machine learning systems: algorithmic bias and sample bias. I explain how algorithmic bias arises from the design of the model, using a credit scoring AI as an example, and how sample bias occurs when training data is not representative of real-world scenarios, like facial recognition systems. I emphasize the importance of recognizing these biases to improve our AI systems. Please take a moment to reflect on how we can address these issues in our projects.

AI decision-making can lead to unintended consequences. This lecture explores AI ethics, safety concerns, and reward hacking, where AI models exploit loopholes in optimization objectives. Learn how to test and prevent these side effects.

Understanding how AI makes decisions is crucial for safety and accountability. This lecture covers explainability techniques such as LIME, SHAP, and model transparency approaches, ensuring AI outputs are interpretable and trustworthy.

Machine Learning (ML) is categorized into different learning paradigms based on how models learn from data. This lecture explores the three main forms of ML: supervised learning, unsupervised learning, and reinforcement learning. Learn how each type is applied in real-world AI systems.

Building a successful ML model follows a structured workflow. In this lecture, we break down the end-to-end ML pipeline, covering data collection, preprocessing, model selection, training, evaluation, and deployment. Understanding this workflow is essential for AI testing.

Data is the foundation of any machine learning system. This lecture covers the steps for preparing ML data, including data collection, cleaning, normalization, and splitting into training, validation, and test sets. Learn best practices to ensure high-quality datasets for AI models.

In this hands-on exercise, you will analyze dataset quality issues such as missing values, class imbalances, and noisy data. Additionally, we will explore different data labeling techniques, including manual, semi-supervised, and automated labeling for training ML models.

The confusion matrix is a fundamental tool for evaluating ML models. In this lecture, we explain true positives, false positives, false negatives, and true negatives, and how accuracy is calculated. Learn why accuracy alone is not always a reliable metric for model evaluation.

Accuracy is not always enough—precision, recall, and F1-score help measure how well a model distinguishes between classes. This lecture breaks down these metrics with real-world examples, followed by an exercise to calculate and compare them.

Different ML tasks require different evaluation metrics. This lecture explores the limitations of accuracy-based metrics and how to select the right performance metrics for classification, regression, and ranking models. Includes an interactive exercise.

AI models must be tested against industry benchmarks to ensure reliability. This lecture introduces benchmarking frameworks for ML, such as MLPerf, ImageNet, and GLUE, and how they help assess model performance across different domains.

Neural networks are the backbone of modern AI applications. In this lecture, we provide an introduction to neural networks, covering their structure, activation functions, and learning process. Learn how artificial neurons work together to recognize patterns in data.

This hands-on exercise walks you through implementing a simple perceptron model, one of the fundamental building blocks of neural networks. You will also train a basic neural network and observe how it learns from data.

Traditional test coverage techniques don't fully apply to AI systems. This lecture explores coverage measures for neural networks, including neuron coverage, layer-wise coverage, and structural testing techniques for deep learning models.

Testing AI-based systems requires a structured approach across multiple test levels, from unit and integration testing to system and acceptance testing. This lecture covers the different levels of AI testing, their objectives, and how they compare to traditional software testing.

AI-based testing is susceptible to automation bias, where testers overly trust AI-generated results. This lecture discusses the risks associated with AI testing, the importance of documentation, and strategies for bias mitigation. A practical exercise is included to help reinforce risk assessment techniques.

AI systems that learn over time introduce unique challenges for testers. In this lecture, we explore how self-learning models evolve, why traditional static test cases may not be enough, and how to develop continuous testing strategies for adaptive AI.

Autonomous AI systems operate without human intervention, requiring rigorous validation to ensure safe and expected behavior. Learn about real-world challenges in testing AI autonomy, including uncertain decision-making, failure recovery, and real-time system validation.

AI-based systems often produce probabilistic outcomes, meaning the same input might not always yield the same result. Learn why non-deterministic behavior complicates testing and how testers can define acceptance criteria, reliability thresholds, and statistical validation techniques.

AI systems integrate multiple layers of decision-making, large-scale data processing, and deep learning models, making testing highly complex. This lecture explores how to handle scalability, dependencies, multi-agent interactions, and emergent behaviors in AI testing.

Unlike traditional software, AI systems lack clear expected outcomes for many scenarios, making test oracles essential. Learn about human, automated, and statistical test oracles, and how they help verify AI outputs when predefined rules do not exist.

Defining test objectives and acceptance criteria is crucial for validating AI models. This lecture covers strategies for setting performance benchmarks, fairness criteria, accuracy thresholds, and robustness goals to ensure AI system reliability.

AI decisions can be opaque and difficult to interpret. Learn how to test for explainability in AI systems, including techniques like SHAP (Shapley values), LIME (Local Interpretable Model-agnostic Explanations), and model visualization to improve AI transparency.

AI models are vulnerable to adversarial attacks, where small, imperceptible modifications to inputs cause incorrect predictions. In this lecture, we discuss data poisoning, evasion attacks, and how attackers manipulate AI systems. Learn how to identify, prevent, and test AI for adversarial robustness.

Pairwise testing is a powerful technique for reducing test cases while ensuring comprehensive input condition coverage. In this lecture, we explore how pairwise (all-pairs) testing can efficiently test AI models, particularly when dealing with high-dimensional inputs.

Back-To-Back testing uses an alternative version of the system to compare the outputs from SUT.A/B testing is widely used in AI applications to compare model versions and optimize performance. This lecture covers how A/B testing is applied to AI-based systems, ensuring data-driven decision-making and improving AI models through real-world testing.

AI systems often lack explicit test oracles, making metamorphic testing an essential strategy. Learn how metamorphic relations help verify AI behavior and detect defects by applying transformations to test inputs and analyzing expected changes in outputs.

Experience-based testing leverages the skills and intuition of testers to uncover unexpected AI behaviors. In this lecture, we discuss exploratory testing, error guessing, and heuristic-based techniques to identify AI model weaknesses and edge cases.

In this hands-on exercise, you will apply exploratory testing techniques and Exploratory Data Analysis (EDA) to an AI dataset. Learn how to analyze data distributions, detect inconsistencies, and identify potential biases before training AI models.

AI-based systems require a mix of traditional and AI-specific testing approaches. This lecture covers how to choose the right test techniques, including statistical testing, rule-based verification, robustness testing, and adversarial testing, based on AI system characteristics.

AI-based systems require unique test environments due to their learning capabilities, autonomous behavior, and interaction with other AI systems. In this lecture, we explore how AI test environments differ from traditional software testing, the challenges involved, and the critical components of a well-designed AI test setup.

Simulated or virtual test environments allow AI models to be tested under diverse conditions, including rare, extreme, and safety-critical scenarios. This lecture explains the advantages of using virtual environments for AI testing, how they enhance repeatability, and their role in cost-effective and scalable AI evaluation.