Testing Machine Learning and GenAI Systems

Explore the machine learning model life cycle from offline data preprocessing, modeling, and evaluation to deployment and online monitoring with drift and latency testing.

Introduce machine learning terminology by defining features as input data and the dependent variable as output, and cover training/test splits, overfitting, underfitting, and QA-driven feature selection.

Demonstrates training a multiple linear regression model on a power plant data set to predict energy output, compares with a random forest regressor, and assesses performance using R2 score.

Master supervised learning by exploring regression and classification models, neural networks, and the role of hyperparameters such as temperature. Learn how historical input-output data trains models to predict new data.

Explore reinforcement learning in ML models by training through environment interaction, using rewards and penalties to maximize outcomes, with examples like AWS DeepRacer and autonomous data center cooling.

Explore top python libraries for ML models, including scikit-learn for diverse algorithms, TensorFlow and Keras for deep learning, PyTorch, XGBoost, and LangChain for large language model apps.

Learn how temperature testing tunes JNI model creativity and determinism, and how token length testing enforces output size limits for production.

Explore zero-shot prompting to test AI models on unseen queries, ensuring graceful handling without task-specific training in customer-facing chatbots.

Test machine learning model APIs using Postman and API keys, with Google Gemini as the demonstration, and explore curl and rest calls for authentication and responses.

Promote responsible AI testing by evaluating safety, ethics, and trust, and perform fairness testing and bias detection with mitigation to achieve neutral results across diverse data.

Practice transparency, explainability, and reliability testing in AI model evaluation, validating that medical decisions are based on clear data inputs and traceable reasoning, while upholding ethical standards across domains.

Conduct societal impact testing through ethical audits and stress tests to ensure AI systems operate equitably and align with long-term societal values.

Test new ai/ml models in shadow mode alongside the current one, diverting traffic to safely expose real data without disruption, then compare canary and A/B results with statistical significance.

Learn to select the right ML model for a business use case by assessing accuracy, bias, and variance, and tuning hyperparameters as a QA professional.

Validate model accuracy with k-fold cross-validation by splitting the training data into k folds, training on k-1 folds, validating the remaining fold, and averaging results with fixed hyperparameters.

Learn how bias and variance shape the bias-variance tradeoff, distinguish underfitting and overfitting, and apply k-fold cross-validation to choose the best model for a data set.

Explore a live model selection demo on a telecom churn dataset, comparing logistic regression, SVM, and random forest using k-fold cross-validation and the bias-variance tradeoff.

Introduction to retrieval augmented generation (rag) as the next step after language models, enabling real-time, domain-specific data by augmenting prompts with a knowledge source and vector store.

Explore the architecture of RAC, detailing ingestion of documents into a knowledge base, converting to embeddings for a vector store, and retrieval via semantic similarity to augment prompts before generation.

Demonstrate retrieval-augmented generation (rag) by turning a tv user guide into a knowledge source, creating embeddings, storing in a vector store, and powering a domain-specific chatbot with semantic retrieval.