ISTQB Artificial Intelligence Testing

Define artificial intelligence and the AI effect; explain narrow, general, and super AI, differences with conventional systems; review AI development frameworks, on-premises hardware, AI as a service, and standards.

Define ai evolution and the ai effect, then compare brute-force chess systems like Deep Blue with learning agents like AlphaZero, highlighting self-play, heuristics, and rule-based limits.

Define narrow, general, and super AI with examples like spam filters, voice assistants, and AlphaZero; discuss whether tools like ChatGPT represent general AI, and explain the idea of technological singularity.

Differentiate AI-based systems from conventional software, highlighting deterministic testable behavior and test oracle for conventional programs, versus probabilistic learning and testing challenges in AI models.

Explore AI technologies, including reasoning techniques, search algorithms, fuzzy logic, and machine learning methods such as neural networks, Bayesian models, decision trees, and SVM, and how AI systems combine technologies.

Explore AI development frameworks for data preparation, algorithm selection, and running AI models on CPUs, GPUs, and TPUs, plus 2021–2025 evolution of MXNet, CNTK, Watson, Keras, PyTorch, scikit-learn, and TensorFlow.

Train in the cloud and deploy to edge computing devices, leveraging CPUs, GPUs, or AI accelerators (ASICs/SoCs) with low precision arithmetic and parallel processing.

Ai as a service offers cloud-based access to models, saving costs under a saas model, while sla limits, bias, and accuracy shape its use with ibm watson and google cloud.

Pre-trained models reduce resource usage and costs by reusing data from related tasks, and transfer learning adapts them—e.g., image classifiers with ImageNet.

Examine how standards and regulations govern AI testing, including GDPR, ISO/IEC JTC1 SC7, and ISO 26262, to mitigate risks and ensure compliant, reliable AI systems.

Define ai concepts and ai effect; differentiate ai-based from conventional systems, cover cpu and gpu hardware, development frameworks, ai as a service, pre-trained models. Explore transfer learning and ai regulations.

Explore quality characteristics for AI based systems, including flexibility, adaptability, autonomy, bias, ethics, side effects, reward hacking, transparency, interpretability, explainability, and safety, with emphasis on understanding rather than testing.

Differentiate flexibility and adaptability in AI: flexibility enables use in environments without changes, while adaptability requires changes to cope with situations; illustrated by robot vacuum cleaner and drone navigation AI.

Define autonomy as a system's ability to operate without human oversight. Compare full and partial autonomy using autonomous cars, factory robots, and ai effect, machine learning, or deep neural networks.

Explains evolution as AI systems improving in changing environments, contrasting it with flexibility and adaptability, and notes learning from decisions or environment changes with safeguards to avoid misalignment.

Explore bias in ai testing, including algorithmic and sample bias, how training data and selection criteria shape outputs, and real-world examples from hiring, lending, policing.

Examine ethics as beliefs guiding AI behavior, noting cultural differences, and summarize OECD's 2019 principles for responsible, transparent, and accountable AI.

Explore side effects and reward hacking in AI systems, examining how goal misalignment can cause negative outcomes and clever shortcuts that bypass intent.

Explore transparency, interpretability, and explainability in AI, contrast black box systems with explainable designs, and examine how users and regulators can trust AI conclusions.

Explore safety in AI systems by examining how complex, non-deterministic, and self-learning decisions challenge harm prevention across medicine, manufacturing, defense, and transportation, including ethical dilemmas and transparency issues.

Summarize chapter two's AI quality characteristics, including flexibility and adaptability, autonomy evolution bias and ethics, side effects versus reward hacking, explainable AI, and safety relations; chapter three introduces practical activities.

Explore machine learning fundamentals, including forms, types, the workflow, and factors for selecting algorithms. Learn to identify and avoid overfitting and underfitting while applying concepts to practice.

Define the three forms of machine learning—supervised, unsupervised, and reinforcement learning—and illustrate how each guides learning through examples, exploration, and rewards.

Master supervised learning with an image classification model: prepare training data, including augmentation, train the model, and test with unseen data to validate accuracy and avoid overfitting.

Learn supervised learning with labeled data, training and testing models to classify inputs or predict numeric values, and understand data labeling, model deployment, and classification versus regression.

Unsupervised learning infers patterns from unlabeled data, forming clusters and associations based on commonalities, with applications in customer segmentation and market basket analysis.

Learn how reinforcement learning trains an agent by interacting with the environment without training data. Guide behavior with reward function design and penalties while avoiding side effects.

Understand the machine learning workflow through an 11-step cycle: define objectives, select framework and algorithm, prepare data, generate and test the model, train and tune, deploy, and monitor for evolution.

Define the machine learning model's purpose with stakeholders, select a framework and algorithm, prepare data, train, evaluate, tune, test, deploy, and monitor for drift.

Explore practical machine learning steps across supervised, regression, unsupervised, association, and reinforcement learning, covering objectives, data prep, model selection, evaluation metrics, tuning, deployment, and integration with non-machine learning parts.

Learn how to choose the machine learning form, from supervised classification and regression to unsupervised clustering and association, with reinforcement learning for interactive environments, using labeled data and training-test splits.

Identify factors guiding machine learning algorithm selection, including functionality (classification, supervised or unsupervised), accuracy versus speed, memory limits, training and prediction speed, interpretability, data availability, testing methods, and trial-and-error comparisons.

Identify how overfitting fits training data yet fails on new data, while underfitting underfits and misses patterns. Explore causes, examples, and how testers detect and mitigate them.

Explore supervised, unsupervised, and reinforcement learning, outline the machine learning workflow, discuss factors in selecting algorithms, and illustrate overfitting and underfitting with a practical exercise.

Explore data preparation, training, validation, and test datasets, and data labeling for supervised learning, highlighting how data quality impacts machine learning models.

Master data preparation in the machine learning workflow, covering data acquisition, pre-processing, feature engineering, labeling, augmentation, and data splitting for training, validation, and testing.

Master a data preparation workflow for AI testing, covering acquisition, labeling with ground truth, cleaning, transformation, augmentation, sampling, and feature engineering with EDA.

Identify training, validation, and test datasets; learn how to split data, choose ratios, and apply methods like k fold cross validation to evaluate and tune models in the ML workflow.

Identify dataset quality issues that bias a model, cause overfitting or mispredictions. Address wrong, incomplete, mislabeled, insufficient, obsolete, duplicate, unbalanced, or irrelevant data, with preprocessing and privacy and security considerations.

Use high quality data to improve model accuracy. Avoid poor data to prevent reduced accuracy, biased models, and compromised predictions from incomplete, mislabeled, unbalanced data and privacy risks.

Explore data labeling for supervised learning, including labeling approaches and common causes of mislabeled data, and highlight the tester's role in ensuring labeling accuracy.

Review the data preparation steps in the machine learning workflow, discuss training, validation, and test datasets, and explain data quality and labeling issues and their effects on models.

Explore chapter five on machine learning functional performance metrics, including the confusion matrix and accuracy and precision. Learn to choose metrics for classification, regression, and clustering while considering their limitations.

Learn how a confusion matrix compares model predictions with ground truth in a spam detection example, defining true positives, true negatives, false positives, and false negatives to evaluate performance.

Evaluate accuracy, precision, recall, and F1-score using spam-filter and medical-test examples to understand when each metric matters and how they balance correct predictions with positives.

Assess a spam detection system with a confusion matrix on 200 emails, computing accuracy, precision, recall, and F1 score from true/false positives/negatives, and note class balance.

Evaluate cancer-diagnosis model with a confusion matrix to derive true positives, false positives, true negatives, and false negatives. Compute accuracy, precision, recall, F1 score, and prioritize recall in medical contexts.

Learn how the ROC curve graphs the true positive rate (recall) and the false positive rate. See how the AUC score summarizes performance across thresholds from 0 to 1.

Explore mean squared error as a key regression metric, showing how squaring errors prevents cancellation and indicates model fit, and learn how R-squared measures the strength of input-output relationships.

Explore unsupervised learning through clustering, measuring intra-cluster similarity and inter-cluster differences to evaluate model quality via the silhouette coefficient.

Examine limitations of machine learning functional performance metrics, which measure model functionality but not non-functional quality like explainability, flexibility, and autonomy; biased data and data pipeline issues distort them.

Select the right machine learning functional performance metrics based on project needs, balancing accuracy, precision, recall, and F1, and choosing AUC ROC for supervised classification or MSE/R2 for regression.

Explore machine learning benchmark suites, including ML Commons, Dawnbench, and ML Mark, and how training and inference times reveal model and hardware performance.

Analyze classification, regression, and clustering metrics from the confusion matrix to accuracy, recall, precision, f1, auc, and rmse, then discuss metric limitations, selection, and benchmark suites.

Explore neural networks as a type of machine learning and their testing, including coverage measures and white-box concepts like statement, branch, boundary value, and state transition equivalence partitioning coverage.

Explore neuron-based coverage measures for neural networks, including neuron coverage, threshold coverage, sine change coverage, value change coverage, and sign coverage, highlighting their activation-valued criteria and current research status.

Explore what neural networks are and what they are composed of, and examine the different coverage measures for neural networks to test them effectively as a software tester.

Explore testing ai-based systems with a focus on specification, requirements and documentation, test levels, data, automation bias, ML model documentation, concept drift, and selecting test approaches.

Design teams face challenges when specifying AI-based systems due to probabilistic, non-deterministic behavior; emphasize high-level goals and constraints with tolerance ranges, addressing test oracle issues and adaptability.

Explore test levels for ai-based systems, emphasizing input data testing and data pipeline quality, and validate the model with performance, non-functional criteria, integration, and acceptance testing.

Explore challenges in test data for ai-based systems, including big data dimensions, aging real-world data, and privacy concerns. Learn techniques to prepare representative, secure test data and avoid pre-processing defects.

Examine automation bias and complacency bias in AI-based systems, testing overreliance on recommendations, insufficient monitoring, decision quality, and representative human input.

Learn how to document AI components with essential information, testing data, performance metrics, ethics and transparency guidelines, and how to validate robustness and adaptability in AI systems.

Explore concept drift as the mismatch between changing environments and an unchanged model, its impact on accuracy, and how to test, detect early, and mitigate via retraining and a/b testing.

Develop a risk-based test strategy for ML systems by combining conventional and AI-specific testing, addressing data quality, pipeline integrity, workflow adherence, performance criteria, explainability, and continuous monitoring.

Specify AI-based systems, apply AI-specific test levels, and use appropriate test data while addressing automation bias. Document ML models, monitor concept drift, and select test approaches for ML systems.