AI Security Bootcamp: LLM Hacking Basics

Explore the top ten security risks facing learning management systems and LLM vulnerabilities, and review the list after watching the video.

Install and set up Docker to begin the hands-on lab in a secure, containerized environment. Download the Docker CLI from the lab manual, install it, and start the Docker engine.

Set up an OpenAI account and generate an api key for upcoming lab exercises. Sign up with a new phone number to receive $5 in free api credits.

Spin up lab container by creating a new directory, downloading and running the Docker image, then access the insecure chat box via the terminal URL.

Explore prompt injection vulnerability in large language models, showing how attackers send malicious inputs to control output and distinguishing direct and indirect prompt injection.

Direct prompt injection, or jailbreaking, overrides the LM prompt with crafted payloads to trigger malicious actions; indirect prompt injection infiltrates data sources with malicious instructions.

Segregate external content from prompts; establish trust boundaries among external sources and plugins; monitor input and output; grant minimum backend access; keep a human in the loop for high-risk actions.

Demonstrate prompt injection techniques to manipulate a chatbot app in the Aida personal assistant app, revealing a secret key and highlighting how instruction handling can leak credentials.

Explore sensitive information disclosure in LMS output, and examine how an attacker deceives the chatbot into leaking confidential information.

Engage in a hacking lab exercise that tests chatbot responses to manipulation, revealing the model owner's identity and passwords when disguises imitate friends or operators.

Explore real-world attack scenarios that arise from insecure output handling, including prompt injection, unvalidated JavaScript payloads causing cross-site scripting, and malicious database queries that can exfiltrate data or erase tables.

Establish trust boundaries between LM external sources, plugins, and downstream functions to prevent insecure output handling, and keep a human in the loop for high-risk actions.

Investigate HTML definitions and tag examples in a chat bot, test for XSS payloads, observe how partial validation triggers popups, and note potential chatbot hallucinations.

Explore real-world attack scenarios arising from overreliance on AI, including accidental plagiarism by alum, misinformation fed to language models causing fake news, and blindly trusting AI-suggested code that introduces vulnerabilities.

Monitor lm outputs for accuracy and bias, cross-check with trusted sources, and break tasks into smaller parts. Implement content filters, clearly label generated content, and provide user warnings about inaccuracies.

Test LLM reliability in a hacking lab by evaluating a personal assistant's accuracy, verifying responses with trusted sources, and highlighting hardcoded credentials and chatbot hallucinations.

Explore how data poisoning acts as an integrity attack by tampering with training data sources to produce false and biased content from chatbots.

Attackers poison training data by injecting falsified or harmful content into LLM training processes. Malicious insiders can inject sensitive data, which may leak through outputs and bias model recommendations.

Verify and attest data sources and bill of materials with machine learning to prevent data poisoning, apply anomaly detection and data sanitization, and sandbox to block sources for your LM.

Create a data source PDF in the AI lab folder from lab manual, then test the chatbot with prompt injection questions to reveal how a polluted data source causes inaccuracies.

Analyze real-world attack scenarios where a language model has read-only repository access. Expose risks from unrestricted plugins, write-enabled database plugins, commands beyond scope, and deletions without user confirmation in production.

Mitigate risks by restricting the Elm agent to plugins, limiting plugin features to prevent open-ended actions, and requiring user authorization via OAuth for LM plugins, with authorization in downstream systems.

Explore how model denial of service exhausts resources, causing outages, crashes, and higher costs as attackers flood chatbots with requests and trigger service disruption.

Explore real-world attack scenarios that delay service by flooding LMS with complex requests, degrading service for users and driving up costs, while attackers manipulate API rate limits to disrupt availability.

Explore a hacking lab exercise that demonstrates how encoded inputs and excessive permissions can disrupt a chatbot service, leading to system downtime.

Implement input validation to enforce defined limits, restrict API requests per user or IP, and cap queued actions in LM-driven systems, while continuously monitoring resource utilization for abnormal spikes.