Threat Modeling for Agentic AI: Attacks, Risks, Controls

Purpose: what agentic AI is, why it fundamentally changes the threat landscape, and establish the architectural baseline for all further threat modeling.

Key Coverage:

• What agentic AI is and why autonomy, memory, and tool use introduce new security risks.

• Why threat modeling is critical for agent based systems compared to classical LLM apps.

• Core components of an agent: planner, memory modules, tool interface, policy engine.

• How agents differ from traditional RAG/LLM systems in behavior, architecture, and attack surface.

• Agent workflows and execution loops: perception → reasoning → action → update.

• Execution graphs, branching paths, recursion, and where failures can cascade.

Activities:

• Course roadmap overview: how all modules fit together for agentic threat modeling.

• Agent architecture walkthrough: visual breakdown of planner, memory, tools, and control boundaries.

Artifact:

• Agent System Reference Diagram

Purpose: Expose the unique and expanded attack surface introduced by autonomous agentic systems and highlight the risks that arise from memory, tools, planning, and multi step behavior.

Key Coverage:

• Memory poisoning vectors that corrupt the agent’s internal state and influence future decisions.

• Unsafe tool invocation patterns and how attackers can misuse toolchains to trigger harmful real world actions.

• Pathways for privilege escalation inside autonomous workflows, including permission drift and unsafe delegation.

• Cascading hallucinations and runaway goal execution that lead to multi step failures and compounding errors.

Activity:

• Agentic attack surface mapping to visualize where and how attackers can influence planner logic, memory updates, and tool interactions.

Artifact:

• Agentic Threat Surface Map.

Purpose: Introduce a structured approach to identifying, analyzing, and mitigating threats specific to autonomous agent architectures.

Key Coverage:

• Extended threat categories unique to agentic systems, focusing on vulnerabilities in memory, planner logic, the tool dispatcher, and the policy engine.

• Common misuse patterns and multi step failure chains that emerge only in agents, including reasoning drift, unsafe delegation, and recursive error loops.

• A complete example of building a threat model for a goal oriented agent with memory, showing how to trace threats through perception, reasoning, action, and update cycles.

Activity:

• Agent threat modeling exercise where learners map threats, attack paths, and mitigations across a full agent workflow.

Artifact:

• Agent Threat Model Template.

Purpose: Provide a structured approach to analyzing and securing the memory layer of agentic systems, focusing on how corrupted or manipulated memory can influence future behavior.

Key Coverage:

• Identifying the primary sources of memory poisoning, including user input, external data connectors, tool outputs, and inherited state from previous reasoning cycles.

• Techniques for sanitizing and validating memory entries before they are stored, ensuring that agents do not internalize harmful or manipulated information.

• Methods for detecting memory drift, tampering, and cross agent contamination, including integrity checks, versioning, and anomaly detection.

Activity:

• Memory threat worksheet for mapping poisoning vectors, evaluating risks, and defining protective controls.

Artifact:

• Memory Integrity Checklist.

Purpose: Equip learners with the ability to analyze, evaluate, and secure the tool layer in agentic systems, focusing on how unsafe tool use can lead to real world harm.

Key Coverage:

• Dangerous categories of tools and high risk capabilities that significantly expand the attack surface.

• Principles of secure sandboxing and permission scoping to limit what agents can do and how far a compromised tool call can propagate.

• Techniques for preventing tool-call abuse, privilege escalation, and unsafe parameter injection through policy controls and schema hardening.

Activity:

• Tool misuse modeling scenario where learners identify threats, analyze escalation paths, and design safeguards for high risk tool interactions.

Artifact:

• Tool Security Checklist.

Purpose: Teach learners how to design strict privilege boundaries and policy layers that prevent agents from performing unauthorized actions or escalating capabilities during autonomous workflows.

Key Coverage:

• Least privilege architecture for agents:
  How to restrict agent capabilities to the minimum required for successful task execution, including scoped permissions, role-based access patterns, and dynamic capability gating.

• Execution isolation and boundary enforcement:
  Techniques for separating execution contexts, preventing cross-component interference, and applying guardrails that halt or redirect unsafe agent actions.

• Oversight mechanisms:
  How to integrate human-in-the-loop validation, supervisor agents, and policy engines that evaluate intent, context, and risk before allowing high-impact operations.

Activity:

• Privilege boundary mapping where learners chart agent permissions, identify escalation points, and design layered oversight and control mechanisms.

Artifact:

• Privilege Control Blueprint.

Purpose: Show how theoretical risks manifest in real systems by walking through concrete incidents involving memory corruption, tool misuse, and reasoning failures. Learners will see how small vulnerabilities evolve into full agentic breakdowns.

Key Coverage:

• Memory poisoning in an agent memory store:
  How corrupted or manipulated memory entries altered future reasoning, shifted intent, and caused the agent to act on false internal state.

• Tool misuse leading to privilege escalation:
  A step-by-step breakdown of how an attacker influenced tool parameters, escalated the agent’s effective permissions, and triggered high-impact actions.

• Hallucination cascade inside a planning loop:
  Examination of how a single hallucinated assumption propagated through multiple planning cycles, creating a multi-step failure chain and compounding errors.

Activity:

• Agent incident reconstruction where learners walk through the timeline of an agent failure, identify root causes, and map how each step contributed to the final incident.

Artifact:

• Agent Incident Map.

Purpose: Set expectations, define audience, and introduce the reference architecture and course structure.

Key Coverage:

• Who this course is designed for: data scientists, software engineers, AI developers, ML engineers, and architects.

• What’s included: conceptual overviews, frameworks, and product categories — not vendor demos or code walkthroughs.

• How to use templates, checklists, and artifacts provided with the course.

• Introduction to the “AI Application Security Reference Architecture” — layers: Model, Prompt, Data, Tools, and Monitoring.

• Brief look at the four categories of AI security products we’ll cover:

  • AI Firewalls / Gateways

  • AI Security Posture Management (SPM)

  • Data Security & Governance Tools

  • Observability & Evaluation Platforms

• Artifact: Course roadmap diagram (visual reference architecture).

Purpose: Understand the evolving attack surface introduced by generative AI systems.

Key Coverage:

• Why traditional cybersecurity doesn’t fully apply to GenAI systems.

• Training-time threats: data poisoning, IP/PII leakage, copyright exposure.

• Inference-time threats: prompt injection, output manipulation, jailbreaks, over-privileged connectors, tool abuse.

• Operational risks: data exfiltration, over-permissioned connectors, hallucination-based attacks.

• Mapping attack vectors across the LLM lifecycle: training → deployment → runtime.

• Early lessons learned from enterprise AI security incidents.

Artifact: “GenAI Threat Matrix” — categorized risk overview.

Purpose: Define how modern LLM-based systems are structured and where controls can be applied.

Key Coverage:

• Common architecture of RAG (Retrieval-Augmented Generation) systems.

• Components: model endpoint, retriever, embedding store, data connectors, tools, orchestration layer, observability layer.

• Identifying trust boundaries and security control points.

• Where to apply policies: input/output filtering, API access, data handling, and logging.

• Comparison between enterprise vs consumer-grade AI architectures.

• Artifact: “LLM Security Reference Architecture Diagram”.

Purpose: Introduce governance frameworks and compliance implications for GenAI deployments.

Key Coverage:

• What AI governance means: principles, policies, and accountability layers.

• AI policies: acceptable use, data handling, retention, escalation.

• Model documentation and evaluation transparency (Model Cards, Data Sheets for Datasets).

• Regulatory frameworks: EU AI Act, NIST AI RMF, ISO/IEC 23894, and OECD principles.

• Defining roles: AI owner, AI risk manager, and AI security engineer.

• Auditability and traceability requirements for enterprise-grade AI.

• Artifact: “AI Policy Starter Template” — outline for internal AI governance policy + data handling matrix

Purpose: Extend traditional threat modeling to generative AI architectures.

Key Coverage:

• Why STRIDE/LINDDUN frameworks need adaptation for LLMs.

• New threat categories: prompt injection, data leakage, tool misuse/abuse, kill-switch,human-in-the-loop points and model drift.

• Practical exercise: building a threat model for a customer-support RAG chatbot.

• Controls mapping: identify, mitigate, monitor.

• Integration with DevSecOps and CI/CD pipelines.

• Artifact: Editable “GenAI Threat Model Worksheet” + worked example.

Purpose: Embed security at every stage of AI product development.

Key Coverage:

• The difference between Secure SDLC and AI-SDLC.

• Secure dataset curation and provenance tracking.

• Model evaluation, safety evals in CI and red-teaming best practices.

• Prompt versioning, change control for chains/graphs. approval, and rollback.

• Secrets management and key isolation in multi-tenant AI environments.

• Artifact: “AI-SDLC Checklist” — security-by-design controls.

Purpose: Explore the first major category of AI security tools — runtime guardrails and firewalls.

Key Coverage:

• What AI firewalls and gateways do: policy enforcement, filtering, monitoring.

• Types of protection:

  • Input filtering (prompt scanning and sanitization).

  • Output filtering (PII masking, toxicity filtering).

  • Tool-call gating and permission enforcement.

• Rule-based vs ML-based vs hybrid approaches.

• Selection criteria: latency, False Positives, policy expressiveness, coverage for tools/functions.

• AI firewall deployment topologies: inline vs API-level.

• Example solutions: Lakera Guard, PromptShield, Guardrails.ai, PromptArmor.

• Artifact: “AI Firewall Evaluation Matrix”

Purpose: Explain how authentication, authorization, and access control protect AI models, APIs, and tools from misuse and unauthorized access.

Key Coverage:

• Why access control is critical for AI endpoints and tool integrations.

• Per-app and per-user API keys, rate limiting, and abuse detection.

• Token scoping and least-privilege permissions for AI tools and connectors.

• Approval flows and human-in-the-loop access for sensitive operations.

• Model/API attestation and response provenance for integrity and traceability.

• Tools overview: Auth0, Azure Entra, and API gateways for policy enforcement and key management.

• Artifact: “AI Access Control Checklist” — key practices for securing AI APIs and identity flows.

Purpose: Introduce continuous monitoring and risk management platforms for AI systems.

Key Coverage:

• What is SPM and why enterprises need it for AI.

• AI asset inventory — models, datasets, connectors, policies.

• Risk scoring and drift detection.

• Policy violations, incident correlation, and reporting.

• Integrations: CI/CD pipelines, ticketing tools, SIEM/SOAR systems.

• Example platforms: Cranium, ProtectAI, HiddenLayer, Aporia.

• Artifact: “AI Asset Inventory Template” — for tracking deployed AI components.

Purpose: Understand how data governance underpins AI security.

Key Coverage:

• RAG data flow — from source repository to model response.

• Data-level access control: ACLs, attribute-based filtering, query-time vs index-time filtering, document tagging.

• Data encryption, anonymization, and tokenization. Encryption at rest/in transit.

• Secure embedding practices — protecting intellectual property and PII.

• How data governance integrates with AI SPM and firewall layers.

• Vendor examples: Pinecone, Weaviate, Qdrant, Databricks Unity Catalog.

• Artifact: “RAG Data Security Checklist” + sample ACL mapping.

Purpose: Understand key categories of security vulnerabilities unique to AI systems and learn practical mitigation strategies.

Key Coverage:

• How indirect prompt injection occurs through external or untrusted content sources, and techniques to detect and sanitize inputs.

• Understanding model inversion attacks and PII leakage — how sensitive information can be reconstructed or revealed from model outputs.

• Identifying supply-chain risks in AI tool wrappers, SDKs, and third-party packages — from dependency tampering to malicious updates.

• Defensive design principles for AI pipelines — input validation, content provenance tracking, and output filtering.

• Secure configuration and patch management practices for AI frameworks and libraries.

• Integration of vulnerability scanning and dependency monitoring into the AI DevSecOps process.

• Artifact: “AI Vulnerability Mitigation Playbook” — examples of common risks, threat patterns, and corresponding countermeasures.

Purpose: Introduce monitoring, evaluation, and telemetry solutions for ongoing AI assurance.

Key Coverage:

• Importance of observability in AI: transparency, reproducibility, accountability.

• What to log: prompts, responses, tool calls, decisions, user feedback.

• Metrics for AI behavior — accuracy, safety, bias, hallucination rate.

• Evaluations as continuous monitoring — quality gates and feedback loops.

• Example frameworks: TruLens, LangSmith, PromptLayer, Weights & Biases.

• Artifact: “Observability Dashboard Blueprint”.

Purpose: Illustrate how enterprises apply AI security controls in real scenarios.

Key Coverage:

• Case 1: Financial services firm using AI firewall + SPM to protect a document assistant.

• Case 2: Healthcare provider securing PHI in RAG-based knowledge bots.

• Case 3: Tech enterprise implementing continuous AI evaluations and risk scoring.

• What worked, what failed, and lessons learned.

• Artifact: “AI Security Implementation Map” — visual summary of combined controls.

Purpose: Help organizations make informed decisions about adoption strategies.

Key Coverage:

• Build vs Buy trade-offs: cost, speed, customization, compliance.

• How to evaluate vendor maturity and security claims.

• Capabilities matrix for firewalls, gateways, SPM, vector DBs.

• TCO, data residency, on-prem vs cloud.

• Key questions for RFP/RFI checklists.

• Integration considerations for hybrid architectures.

• Future trends — convergence of AI gateways, SPM, and observability layers.

• Artifact: “Vendor Evaluation Questionnaire”.

Purpose: Consolidate learning by assembling an end-to-end AI security control map.

Key Coverage:

• Map threats → controls → products.

• Choose appropriate controls for each layer of LLM/RAG architecture.

• Build an AI security roadmap for your organization (30/60/90-day plan).

• Identify continuous monitoring and compliance processes.

• Artifact: “AI Security Control Stack Template”