LLM Testing Masterclass: Software QA for AI Language Models

Write a Python REST client to support the test framework for LLM QA. Connect to /api/generate, send prompts with model and temperature, and parse latency and tokens.

Explore the full test coverage map for llm qa, spanning functional and non-functional tests—from basic response and instruction following to sentiment analysis, code and sql generation, and hallucination detection.

Use a pytest fixture to initialize a test client for the tiny llama model, skip if unavailable, and perform a functionality test that calls generate and checks a non-empty response.

Continue building basic functional tests, including instruction following, simple qa, and multi-turn context retention, and preview content processing and generation test cases for the next lecture.

Explore functional testing of content processing and generation for AI language models, including writing prompts, fixtures, regex-based format validation, and JSON output verification.

Explore context learning capabilities in functional testing by validating logical consistency with equivalent questions like what is the capital of Japan and Tokyo is the capital of which country.

Perform a functional test of context learning by evaluating a language model’s error handling through command-line prompts, and write assertions to detect nonsensical or undefined responses.

Cover non-functional performance testing for llm by limiting output tokens, running three requests, and calculating throughput, tokens per second, and requests per minute.

Evaluate robustness in non-functional testing by validating a language model's handling of input variations and adversarial prompts, ensuring Paris appears across case changes.

Explore non-functional hallucination detection in llm testing for lms by crafting fictional prompts and evaluating responses. Write test logic to detect and avoid hallucinations from tiny llama models.

Leverage the deep eval framework to treat the language model as a judge, evaluating target model responses with criteria, optional ground truth, and metrics like relevance and coherence.

Explore the deep eval framework to rewrite native Python test cases for LMs as framework-based tests, leveraging open source evaluation metrics and pytest-like validation.

Build an OpenAI test client wrapper to replace a local LLM, wiring an OpenAI API key, and using the completions API with chat-style messages while measuring tokens and latency.

Learn functional testing with the OpenAI test client, focusing on LM-based evaluation metrics like answer relevancy and custom metrics, using natural-language criteria and GPT-4 for evaluation.

Learn to configure evaluation parameters for language model tests, set a 0.7 pass threshold, and build instruction-following test cases that verify prompts and responses like listing colors.

Learn functional testing with DeepEval for basic Q&A, using expected output and evaluation parameters, thresholds, and an evaluation model to verify that Paris is the capital of France.

Explore functional testing with DeepEval for content processing and generation, using criteria-driven, llm-powered verification of list and json outputs, and crafting precise prompts for automated evaluation.

Master the context learning test cases with DeepEval, evaluating logical consistency and error handling using defined criteria, combined prompts, and adjustable thresholds for non-deterministic LLM QA.

Apply DeepEval’s hallucination metric within the DPI framework, supplying a context and evaluating model outputs against it to detect hallucinations.

Explore retrieval augmented generation (RAG) to fuse knowledge retrieval with AI generation, enabling up-to-date answers from external knowledge bases and reduced hallucinations.

Explore the contextual relevancy metric from the Deepwell library for evaluating retrieval context. A shoe store example shows how relevant context guides LLM responses in a rag pipeline.

Explore how a rag application uses embedding models and a Pinecone vector database to retrieve context and answer customer shoe queries with a language model.

Build and test a shoe store rag application by setting up a pinecone index, generating embeddings with OpenAI, populating knowledge base, and running queries and metrics against a language model.

Prepare documents by embedding text and collecting ids and metadata, then upsert vector data into Pinecone to build a shoe store RAG knowledge base.

Continue building the shoe store RAG application by implementing the context retrieval function, embedding queries, querying the Pinecone index, and extracting metadata as context documents for answer generation.

Build a prompt from user query and retrieved context, call the chat completions api, and deliver a helpful, context-based shoe store answer.

Learn to build and test a shoe store rag app by writing test cases, retrieving context with Pinecone and OpenAI keys, generating answers, and evaluating context relevance with GPT four.