Testing Machine Learning and GenAI Systems

Explore the machine learning model life cycle from offline data preprocessing, modeling, and evaluation to deployment and online monitoring with drift and latency testing.

Introduce machine learning terminology by defining features as input data and the dependent variable as output, and cover training/test splits, overfitting, underfitting, and QA-driven feature selection.

Master supervised learning by exploring regression and classification models, neural networks, and the role of hyperparameters such as temperature. Learn how historical input-output data trains models to predict new data.

Explore reinforcement learning in ML models by training through environment interaction, using rewards and penalties to maximize outcomes, with examples like AWS DeepRacer and autonomous data center cooling.

Promote responsible AI testing by evaluating safety, ethics, and trust, and perform fairness testing and bias detection with mitigation to achieve neutral results across diverse data.

Practice transparency, explainability, and reliability testing in AI model evaluation, validating that medical decisions are based on clear data inputs and traceable reasoning, while upholding ethical standards across domains.

Conduct societal impact testing through ethical audits and stress tests to ensure AI systems operate equitably and align with long-term societal values.

Learn to select the right ML model for a business use case by assessing accuracy, bias, and variance, and tuning hyperparameters as a QA professional.

Validate model accuracy with k-fold cross-validation by splitting the training data into k folds, training on k-1 folds, validating the remaining fold, and averaging results with fixed hyperparameters.

Demonstrate retrieval-augmented generation (rag) by turning a tv user guide into a knowledge source, creating embeddings, storing in a vector store, and powering a domain-specific chatbot with semantic retrieval.