OSCP for AI: The GenAI Security Sandbox

Learn how direct prompt injection creates a vulnerability in a translation bot by concatenating system instruction with user input, and apply the fix through separation of roles.

Demonstrate a prompt injection vulnerability in a translation bot by mixing system and user input and overriding translations. Show how separating system and user instructions prevents the attack.

Explore direct prompt injection defense by implementing the sandwich defense that splits system and user instructions, translating untrusted input into French and reminding the model not to trust user data.

Expose insecure output handling in llm apps, showing how shell-based command execution enables remote code execution and prompt injection; implement guardrails, remove shell, and validate against a whitelist.

Explore insecure output handling vulnerabilities by crafting LLM-generated Linux commands that are executed via shell, exposing remote code execution and local file inclusion risks.

The video demonstrates how insecure output handling in an LLM enables a reverse shell through a command chaining attack, highlighting a blind execution vulnerability.

Implement a strict allowlist to map intents to predefined keys like list files, date, and who am I; execute commands as direct processes with shell disabled for robust security.

Analyze insecure plugin design in a vulnerable app that blindly executes LLM commands, creating a confused deputy risk and phishing potential. Learn validation and human-in-the-loop checks to block abuse.

Explore insecure plugin design by exploiting a vulnerable email system that blindly executes a send_email trigger from an LLM output, lacking validation and enabling internal spear phishing and prompt-injection attempts.

Explore defense against insecure plugin design with a strict validation layer, domain whitelist and human-in-the-loop approval to prevent data leakage and the confused deputy problem.

Demonstrates a multi-layer defense against sql injection by restricting to select statements and blocking dangerous keywords. Enforces a one-statement rule and prevents destructive queries before reaching the database.

This lab showcases a multi-layer defense against SQL injection in AI systems by enforcing one-statement-per-call, narrowing to select statements, and blocking drop and delete commands.

Explore how LLM-generated cross-site scripting arises when user prompts generate HTML that is rendered unsafely, and learn safe rendering and sanitization techniques to prevent XSS.

Explore cross-site scripting vulnerabilities by using streamlit components to demonstrate how unfiltered ai-generated html or javascript can execute in an iframe, enabling reflected and zero-click XSS and defacement risks.

Explore how a denial of service targets a large language model, exhausting cpu, gpu, and memory through unbounded generation. Enforce max_tokens and timeouts to protect resources.

Implement a hard 50-token limit on the LLM's outputs to act as a circuit breaker, truncating responses and defending against denial-of-service and resource-exhaustion attacks while keeping API costs predictable.

Explore a PII vulnerability where a hard-coded system prompt exposes private data and prompt leakage, and the LLM returns unredacted emails and phone numbers through simple queries.

Explore the dangers of insecure history management in llm apps, where session state stores chat history and prompt injection taints context, highlighting sanitation and context-window defenses.

Demonstrate context poisoning in a chatbot by abusing an unvalidated persistent history, using persona injections and system overrides that persist across sessions.

Defend against context poisoning by enforcing a stateless, ephemeral design. Each interaction constructs an isolated messages list using the immutable system prompt and current user input, flushing memory after generation.

Explore RAG poisoning and knowledge integrity by examining a vulnerable app that lets anyone rewrite the company wiki. Learn how strict ACLs and authentication prevent data poisoning in AI systems.

Learn how data poisoning in retrieval augmented generation taints a shared knowledge base wiki, causing the AI to spread misinformation and malicious links, with safeguards against unauthorized edits.

The patch replaces the mutable state with a static immutable constant, trusted as a read-only knowledge base, using IDs like ID 101 and 102 for work hours and remote policies.

Excessive agency in AI power agents creates an infinitive loop that wastes CPU and memory, risking denial of service. Fix by using bounded for loops, global timeouts, and plan validation.

Demonstrate excessive agency vulnerability with an autonomous AI agent in a loop. The lab shows denial-of-service risks and how TTL limits prevent endless execution.

Replace the infinite while loop with a for loop and a max loop limit of five iterations to enforce bounded agency, ensure predictable resources, and trigger human control on termination.

Learn how llm overreliance leads to hallucinations and supply chain attacks, and how to verify ai suggestions by checking pypi.org, project pages, and documentation before installing.

Demonstrates training data poisoning by tainting an HR chatbot's fine-tuned model with a hidden severance trigger word, causing a backdoor to override outputs in production.

Explore multimodal injection, where image viruses embed hidden prompts in pixels to override AI behavior, and learn defenses using OCR scanning and image regeneration to sanitize uploads.

Learn how a fixed multimodal injection defense uses an OCR security layer as a mandatory checkpoint before images reach the multimodal llm, blocking prompt injections.

Learn how attackers exploit a vulnerable API with reach return to clone proprietary models, and how to harden the API to return only a final label and apply rate limiting.

Investigate model theft and extraction vulnerabilities where an API reveals full probability scores for public, internal, and confidential categories. Emphasize returning only the final label to prevent leakage.

Explore supply chain attacks in AI development, focusing on the pickle bomb vulnerability in downloaded model weights from HuggingFace, the risk of remote code execution, and safe tensors.

Explore membership inference attacks that reveal whether a person’s records entered model training by memorizing prompts. Learn how differential privacy and data sanitization guard against such leaks.

Explore privacy and PII leakage in a language model trained on unredacted data, highlighting lack of filters and guardrails and the need for RBAC and data sanitation.

Protect privacy by adding a PII scrubbing layer at the output stage, using DLP filters, regex and NER to redact credit cards, SSNs, addresses, and medical history.

Explore audio jailbreaking vulnerabilities in Siri-like voice assistants, where speech-to-text pipelines and brittle keyword filters can be bypassed, then deploy phonetic similarity and intent classification defenses.

Examine how audio prompt injection exploits transcription models in a multi-model app, revealing an attack surface expansion when audio is not sanitized, and explore defenses against audio-based prompt hijacking.

Defend against audio hijacking with a sandboxed transcription layer that wraps output in XML, uses a strict system prompt to summarize only within tags, and blocks hijack keywords.

Install garak, the Generative AI red team and assessment kit, an open source vulnerability scanner for large language models, using mini conda to create a Python 3.10 environment and llama.

Explore Garak's command-line basics, learn how to view syntax, models, and reporting options; inspect probes, detectors, and buffs; review config, and try interactive mode to test prompts in real time.

Test how Garak interacts with llama2-uncensored to probe jailbreaks like DAN, exposing safety bypasses. Evaluate attack success within the GenAI security sandbox.

GizCard is an open-source quality assurance framework for testing AI models, automating detection of prompt injection, data leakage, harmful content, bias, and hallucinations with CI-CD integration and the LLM scanner.

Create and activate a conda environment named gizguard_lab with Python 3.10, install GizGuard via pip, then install pandas, langchain, and langchain-community, and verify the installation with version 2.18.0.

Configure Giskard with llama and manage LangChain version differences. Set up an OpenAI compatible API, instantiate a gamma judge with a llama victim and seed prompts for security scanning.

Explains the master scanner workflow for auditing an llm, including fast or deep scans, llama3 model configuration, a timestamped html report, and an interactive menu with 1–7 vulnerability mappings.

Run the master score scanner in fast jailbreaking mode against Lama three using the gamma judge, set up in the discard environment, and review the generated html report for evaluation.

Examine indirect prompt injection in a GenAI security sandbox, showing how external data sources like PDFs and websites can manipulate an LLM via third-party content.

Explore context leak and system prompt leakage by testing Gemma versus Llama3 with the gist card lab master scanner in fast mode, to reveal whether hidden instructions are exposed.

Learn how jailbreaking GenAI models works by crafting prompts to bypass safety filters, compare Mistral and Llama 3 defenses, and analyze Gemma-backed jailbreak reports for intent and safety implications.

Compare jailbreaking, indirect prompt injection, and context leak between mistral and llama3 using a master scanner and OSCP reports; conclude that llama3 performs better overall with no disclosed issues.

Set up a pirate lab conda environment on score lab, install pandas, python-dotenv, and pyrite, then run the pyrite master script to audit AI models across categories.

Practice prompt-based jailbreak attacks in the PyRIT lab, testing guardrail bypass on the 5-3 mini LLM using vectors like base64, rot13, lead speak, and join hyphen.

Demonstrate direct prompt injection in LLMs by overriding system prompts with user prompts, using the PyRIT lab to assess vulnerabilities and review injection reports.

Examine hallucination and misinformation in AI using pyrite, detailing how LLMs generate false text and how reports reveal spread of misinformation across five encoding methods, addressing two questions.

Compare 53 and llama 2 uncensored by running four attacks on each, generate an html risk scorecard, and evaluate jailbreak, direct prompt injection, hate speech, misinformation, and hallucination risks.

Learn how to extend the PyRIT master by adding more prompts across categories like jailbreak and injection, improving the Python script's security audit with comma-separated entries and multiple encoding variants.

Add Gwen2 as a new LLM to the pyrite master script by updating the models dictionary and installing locally, then verify the sha-256 digest.

Adopt an adversarial operator role to tackle five CTF challenges in the OSCP for AI security lab, exploring LLM vulnerabilities through the chat interface and earning quiz codes.