A Deep Dive into LLM Red Teaming

Explore the ten OWASP vulnerabilities in text generation, including prompt injection, insecure output handling, data poisoning, and model theft, and learn defenses for LLM applications.

Explore Google's secure AI framework (Safe) for end-to-end security across data, infrastructure, model, and application, detailing risk types, mitigation controls, and a unified risk map for trustworthy deployments.

Adapt red team tactics for generative AI security as ML evolves, emphasizing black box testing, data handling, and four core components: model, data, application, system.

Explore the model component of a machine learning system, including weights, biases, and training, and examine risks like model poisoning, prompt injections, and model extraction.

Explore how the data layer, from training data to inference inputs, can be manipulated through data poisoning and backdoors, risking data leakage, bias, and GDPR implications.

Explore how ml-based systems inherit application-layer security risks from traditional software, including injection attacks, insecure authentication, weak input validation, cross-site scripting, and threats to confidentiality, integrity, and availability.

Explore the security of the system layer behind ML deployments, focusing on misconfigurations, insecure model deployments, and threats like denial of service, password spraying, vulnerability scanners, and exposed admin interfaces.

Explore how system prompts and user prompts shape llm behavior, reveal prompt injection risks, and how multi-turn prompting and domain guidance help maintain safe, focused responses.

Watch a Kali Linux demo showing prompt injection as a critical vulnerability that inflates CV evaluations by manipulating the OpenAI API, illustrating the need to secure AI systems.

Demonstrates prompt injection on a large language model with Gandalf, guiding through eight levels from direct prompts to indirect tricks to reveal a password in a red-teaming demo.

Analyze insecure output handling in llms through a Portswigger lab demo on indirect prompt injections and XSS, illustrating how attacker can compromise user accounts and manipulate password resets.

Conduct llm red teaming focused on training data poisoning and prompt injections, using Garrick to reveal vulnerabilities for security report on toxic outputs, dan-style attacks, phishing, malware, hallucinations and xss.

Explore how a denial of service attack tests an LLM API by flooding requests and complex prompts, and learn defenses like rate limits and semaphore-based concurrency.

Explore LLM red teaming by simulating a typosquatting attack on the Python requests library, revealing cookie theft and session hijacking risks in software supply chains.

Explore how large language models can disclose sensitive information and how to mitigate PII leaks through data sanitation, input-output safeguards, and secure training practices in an LLM red-teaming context.

Examine insecure llm plugin design that enables unauthorized access via raw sql prompts in the User Fetcher plugin, and apply mitigations with parameterized queries and least privilege.

Explore the red team lesson of excessive agency vulnerability in an LLM-based customer support system, showing how limited permissions, rate limiting, and a human-in-the-loop CMS prevent unauthorized refunds.