Penetration Testing for LLMs

Discover how generative AI creates new content via machine learning, producing text, images, and audio with techniques like GANs, and explore applications and ethical concerns.

Explore what a large language model does: it predicts the probability of the next token from context, not real intelligence, illustrated with how you feel.

Define prompts as the input users provide to large language models to generate outputs, and explain crafting prompts with context and formats to improve results.

Explore how AI models train on text, images, and other data to form foundation models, then adapt them for specific use cases like security analysis and incident summarization.

Explore the LMS architecture from user prompts through the LM application, including application services, automation agents, the model, data sources, and plugins, with OWASP top ten security surfaces.

Explore how adversaries use AI to craft misinformation, fake images and news, personalized phishing at scale, and to research vulnerabilities and generate exploits.

Explore Microsoft's responsible AI approach, covering fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability in AI systems and their human impact.

analyze the shared responsibility model for AI in the Microsoft cloud, detailing duties between Microsoft and customers across IaaS, PaaS, and SaaS, including user training and identities and access management.

Penetration testing is a simulated cyber attack to identify vulnerabilities before real attackers exploit them, using techniques such as the Mitre attack and the Mitre Atlas framework.

Penetration testing detects security weaknesses, informs risk management with real-world vulnerability data, validates controls, and guides proactive security improvements through remediation insights and resilience building.

Red teaming provides an end-to-end security assessment using adversary-like ttps, while penetration testing targets specific vulnerabilities within a defined, time-bound scope.

Perform penetration testing for gen ai platforms to uncover vulnerabilities that enable model inversion and data exfiltration, assess privacy compliance, and prevent unauthorized access to sensitive data.

Identify prominent threat vectors for genai applications, including prompt injection, data leakage, overreliance, data poisoning, supply chain risks, insecure plugins, jailbreaks, and model vulnerabilities across data, usage, application, and platform.

Explore the ai pen testing process for llms, starting with planning and preparation, then reconnaissance, scan and enumerate, vulnerability assessment, exploitation, post exploitation, reporting, remediation, and follow-up with stakeholders.

Define objectives and scope for a penetration test, identify stakeholders and permissions, gather resources, set a detailed timeline, and conduct a risk assessment to anticipate impact on systems.

Conduct reconnaissance by gathering public information on the target llm and its environment, analyze apis and endpoints, study user interactions, review documentation, and identify potential attack vectors.

Conduct API endpoint scanning and input analysis to map data flows and enumerate backend services and dependencies in the LLM, then perform a configuration review to identify vulnerabilities and misconfigurations.

Conduct automated testing with tools like pirate and the Python risk identification tool, perform manual inspections, analyze outputs, test for injection flaws, and prioritize vulnerabilities by severity and exploitability.

Explore the exploitation phase by developing and validating exploits for identified vulnerabilities, simulating controlled attacks, testing privilege escalation and indirect prompt injection, and analyzing impact on large language models.

Assess data exfiltration risk by attempting to extract LMS data in post-exploitation. Demonstrate persistence with backdoors and C2 infrastructure, map LMS internal structure, enable lateral movement, and clean up artifacts.

Document detailed findings of vulnerabilities and exploited weaknesses, perform risk and impact analysis, rank risk by severity and exploitability, and provide actionable recommendations and executive summary for non-technical stakeholders.

Outline remediation strategies, mitigation actions, and implementation support; emphasize retesting, ongoing monitoring, and lessons learned to improve future penetration tests.

Explore the MITRE attack framework, its tactics, techniques, and procedures, and learn how threat-informed defense guides coverage, with a Sentinel case study.

Map the pyramid of pain to ATT&CK by aligning tactics, techniques and sub-techniques with adversary operations, focusing on TTPs rather than hash values, IP addresses, or domain names.

Examine the three big matrices—enterprise, mobile, and ICS—with submatrices for Windows, Linux, Mac OS, Azure AD (intra ID), Office 365, Google Workspaces, networks, containers, and mobile platforms.

Explore the Mitre attack framework’s tactics and how adversaries pursue objectives like reconnaissance and initial access. Describe how tactics cover stages from reconnaissance to impact.

Explore the Mediatech framework's techniques, showing how attackers perform across reconnaissance, initial access, persistence, and defense evasion with examples like active scanning, brute force, internal spearphishing, and encrypted C2 channels.

Explore sub techniques in detail, mapping 424 sub techniques to tactics and techniques, with examples like vulnerability scanning, spear phishing attachments, dll injection, and password spraying.

Explore tactics, techniques, and subtechniques in the attack framework, including motivation behind actions and how execution, command and scripting interpreter, and Python enable adversaries to achieve objectives.

Identify and onboard data sources to collect telemetry and detect attacker activity, focusing on network traffic and logs from the web application firewall for vulnerability scanning in Mitre attack framework.

Identify mid-air detections to guide a high level detection strategy for TTPS, focusing on reconnaissance, active scanning, and vulnerability scanning via web application firewall logs and CIM alerts.

Apply preventative configurations to reduce the attack surface and minimize data exposed to external parties, and establish privileged account management to mitigate privilege escalation.

Explore how threat groups show related behavior and are tracked by vendor-specific names, such as apt41, Fancy Bear, and Midnight Blizzard.

Explore software as the tools and malware adversaries use, linked to techniques, groups, and campaigns, including built-in and publicly available software like PowerShell.

Campaigns orchestrate intrusion activities over a defined period with common targets and objectives, usually by nation-state actors or organized crime groups, such as the Ukraine power grid attack.

Explore how groups, tactics, objectives, and motivations interrelate to form a campaign, and how techniques and sub techniques enable targeted actions. See how data sources empower detections that counter adversaries.

Explore the MITRE ATT&CK enterprise matrix in a browser demo, covering tactics, techniques, sub-techniques, mitigations, detections, data sources, and CTI context.

Explore mid-air Atlas, a framework that provides techniques, tactics, and procedures for AI to defend against known attack methods in generative AI for CTI purposes.

Explore Atlas tactics—14 attack-related tactics including initial access, ML models access, and machine learning attack staging—bridging concepts with the attack framework to understand adversary objectives.

Explore techniques within the Atlas and attack frameworks, showing how adversaries perform attacks in AI and ML contexts, including prompt injection, ML attack staging, and ML model backdoors.

Explore Atlas techniques in the Mediatheque framework, including prompt injection and its direct and indirect subtechniques. Examine examples like machine learning model access and backdoor poisoning of models.

Combine tactics, techniques, and sub techniques to map attacker logic. Apply this to ChatGPT with a tactic of initial access and a technique of prompt injection.

Atlas provides mitigations to reduce the attack surface with preventative configurations; some techniques have mitigations, others do not, such as encrypting information to protect ML IP.

Explore Atlas case studies, showing how production AI systems face evasion, poisoning, replication, and flaws, across ML as a service, cloud, edge, with diverse personas.

Analyze a 2020 case where the Microsoft AI red team disrupted an internal Azure service, blending traditional attack techniques with adversarial ML evasion and Atlas framework insights.

Case study II PoisonGPT demonstrates how a poisoned LLM can return false facts and spread misinformation after researchers uploaded a poisoned model to Hugging Face, highlighting supply chain risks.

Explore the May 2023 indirect prompt injection attack on ChatGPT plugins, revealing a privacy leak where an attacker exfiltrated chat history and PII by compromising a chat session.

Explore the Atlas matrix alongside the attack framework, navigate tactics and techniques with an embedded navigator, compare detail and limitations, and review case studies, mitigations, and blue‑red team threat modeling.

Learn about the open worldwide application security project (OWASP), its top ten risks for web apps, APIs, and large language models, and open-source tools like OWASP ZAP.

Explore the updated OWASP top ten for large language models in 2025, covering prompt injection, sensitive information disclosure, supply chain, data and model poisoning, misinformation, and unbounded consumption.

Explore prompt injection in large language models, distinguishing direct and indirect attacks, illustrating jailbreak-style prompts and hidden data in documents, and analyzing risks like data leaks and unintended actions.

Explore how large language models disclose sensitive information, including PII, financial records, and proprietary data, via misconfigurations, prompt injections, and unintentional exposure.

Examine supply chain risks in large language models, including third-party models, data, and tools, and learn how vulnerabilities and biases threaten integrity and safety.

Explore data and model poisoning threats to large language models, including training data manipulation, backdoors triggered by inputs, and supply chain vulnerabilities that degrade performance and spread misinformation.

Explore improper output handling in LLMs, focusing on validation and sanitization to prevent cross-site scripting, SQL injection, and remote code execution, including indirect prompt injections and poor encoding.

Limit excessive agency by capping LM agents' functionality, permissions, or autonomy, to prevent harmful actions such as malicious prompt injections, hallucinations, privileged escalation, and lack of human oversight.

Explore how system prompts leak api keys and credentials, and how prompt injection can bypass controls or cause disclosure.

Explore vector and embedding weaknesses that expose sensitive data. Examine how unauthorized access, context leaks, embedding inversion attacks, and data poisoning undermine llm security.

Explore how misinformation from large language models can generate false, misleading outputs with security, legal, and reputational risks. Learn to recognize hallucinations, verify claims, and reduce overreliance on model outputs.

Unbound consumption causes excessive inferences that deplete resources, degrade performance, and can trigger denial of service, wallet exhaustion, or costly model extraction in cloud LLM deployments.

Set up a ready-made, vulnerable LLM lab using Portswigger's pre-built model, no VM required, and explore four OWASP top ten vulnerabilities with practical exploitation examples.

Demonstrate indirect prompt injection in a lab environment by guiding an authenticated user through API calls to delete a user account in a web shop.

Learn to identify insecure output handling in an LMS lab that combines indirect prompt injection and cross-site scripting to test and demonstrate account deletion risks.

Demonstrates exploitation of supply chain vulnerabilities in large language model apis, achieving remote code execution to delete a file via the newsletter subscription api.

Examine excessive agency in llms by probing available apis, including a raw sql demo, which enables deleting a user like Carlos and exposing security risks.

Celebrate finishing this course, leave a review, and connect on LinkedIn or X, while exploring discounts and subscribing to Azure and cybersecurity newsletters.