Non Functional Testing for LLM, Chatbots and AI Models

Get a brief introduction on what are the main components of AI

Understand how LLMS and foundation models are part of the AI Science field.

How NLP actually makes the AI more human.

Understand what is machine learning and how algorithms make the core of AI

Understand the basics concepts around supervised Machine Learning

Gain basic understanding of Unsupervised ML and Clustering

In this lecture you will get a basic idea of how Reinforced Learning is working together with ML Algorithms

In this material you will understand how critical good quality training data actually is.

Understand what are the main areas in software testing

Understand what PerspectiveAPI can offer in matter in toxicity and censorship;

Step by step demo how to obtain a ChatGPT API Key

See a live demo how we will implement an API call to Perspective API

Github Code: calculator/src/test/Perspecttive_api at master · danteachqe/calculator · GitHub

Understand how Perspective API And ChatGPT work together to create an automated testing framework for Toxic content.

In this lecture you will learn the basics of adversial testing as well as the red and blue security teams.

Learn how prompt injection works by understanding, direct and indirect prompt injections techniques as well as some examples and how to defend against them.

Fuzz testing, also known as fuzzing, is a software testing technique used to discover security vulnerabilities and bugs by inputting large amounts of random or semi-random data (often referred to as "fuzz") into a program or system.

Understand DOS attacks such as: Prompt flood, API Exploitation, Context Window Exploit and other

Understand what is poisoning attack, how to mitigate it and some examples.

Links to material reference:

https://www.businessinsider.com/tesla-hackers-steer-into-oncoming-traffic-with-stickers-on-the-road-2019-4

https://bair.berkeley.edu/blog/2019/08/13/memorization/

https://chatgpt.com/share/456d092b-fb4e-4979-bea1-76d8d904031f
https://www.researchgate.net/figure/Adversarial-examples-using-PGD-with-and-with-noise-constraint-of-on_fig1_350132115

A poison attack involves injecting malicious data into a training dataset, causing the model to learn incorrect or harmful behaviors. The attacker subtly alters training inputs so that the model produces incorrect outputs during real-world use.

Understand the lesser known aspects of non-functional Testing for Conversational AI

1. Description: How well does the model align with ethical guidelines and avoid generating harmful, biased, or inappropriate content?

2. Testing Aspects:

   • Does the model refrain from generating offensive or dangerous outputs (e.g., hate speech, misinformation)?

   • How well does it handle sensitive topics without being exploitative or biased?

1. Description: The ability of the model to provide understandable reasoning for its outputs.

2. Testing Aspects:

   • Can the model explain why it provided a certain answer or the process it used to arrive at a decision?

   • Is there transparency in how it weighs different inputs?

1. Description: How well the model adapts to different types of user interactions, including ambiguous, contradictory, or incomplete queries.

2. Testing Aspects:

   • Can the model maintain coherent conversations across multiple queries or threads?

   • How does it handle user queries that contain slang, mixed languages, or unusual input formats?

1. Description: The ability to retain and appropriately use context across long conversations or interactions.

2. Testing Aspects:

   • How effectively does the model remember previous inputs in a long interaction without getting confused or providing irrelevant responses?

   • Does the model avoid "forgetting" context, or improperly continuing it when it's no longer relevant?

1. Description: How well the model generates novel or creative content when prompted.

2. Testing Aspects:

   • Can the model create new stories, jokes, or analogies that are both original and coherent?

   • Does it avoid repetitiveness or overfitting common outputs when asked for creative work?