Strategies in Secure AI Systems: From GenAI and Agentic AI

This section addresses autonomous AI agents that execute multi-step workflows, interact with external systems, and maintain memory, introducing unique threat vectors beyond traditional LLM risks including goal hijacking, tool misuse, inter-agent deception, and memory poisoning. The OWASP Top 10 for Agentic Applications (ASI) framework provides security controls across the agent lifecycle, covering ASI01 Agent Goal Hijack (manipulating objectives through untyped inputs), ASI02 Tool Misuse (agents abusing legitimate tools for unintended purposes), ASI03 Identity and Privilege Abuse (exploiting delegation chains and cached credentials), ASI04 Agentic Supply Chain Vulnerabilities (compromised dynamically-sourced components), ASI05 Unexpected Code Execution (exploiting code-generation features for RCE), ASI06 Memory and Context Poisoning (corrupting persistent memory to bias future decisions), ASI07 Insecure Inter-Agent Communication (lack of authentication and integrity in multi-agent systems), ASI08 Cascading Failures (single faults propagating across autonomous agents into system-wide harm), ASI09 Human-Agent Trust Exploitation (leveraging automation bias to influence user approvals), and ASI10 Rogue Agents (compromised agents acting maliciously with persistent unauthorized behavior). Mitigation strategies span the development lifecycle including system prompt hardening with explicit constraints, memory security design with least privilege and PII redaction, human-in-the-loop controls for high-impact actions, mandatory sandboxing for tool and code execution, I/O validation using AI guardrails, and comprehensive monitoring with behavioral detection and immutable audit logs.

Effective AI governance requires establishing an AI Governance Board led by a Chief AI Officer with diverse stakeholder representation, along with a formal RACI chart defining accountability for AI risks. Organizations must define risk tolerance, create comprehensive AI policies, and publish acceptable use guidelines for generative AI tools.

The OWASP LLM Governance Checklist provides a security framework covering model security, data governance, access control, output monitoring, adversarial robustness, legal compliance, and lifecycle management. Operational governance focuses on data management policies, regular compliance audits aligned with frameworks like NIST AI RMF and GDPR, agentic systems oversight, and comprehensive audit logging.

Legal and accountability governance addresses external compliance requirements and liability. Organizations must review AI usage against evolving regulations, verify vendor compliance, and establish clear indemnification provisions. Special attention to bias concerns in HR applications requires documented training methods and proper consent mechanisms for sensitive information collection.

Organizations face a critical visibility challenge: their AI footprint extends far beyond sanctioned systems like corporate chatbots and code copilots. Shadow AI—including third-party apps with AI features, insecure browser plugins, unvetted open-source models, and employee use of public GenAI tools—poses substantial risks including data exfiltration, operational failures, intellectual property loss, and compliance failures. An AI Asset Inventory serves as a comprehensive catalog tracking model versions, data provenance, ownership, deployment status, and risk scores for every AI asset. This single source of truth is essential because organizations cannot govern what they cannot see.

To transform visibility into strategic defense, organizations should implement a three-pillar framework: GOVERN by establishing clear ownership and policies through an AI Governance Board; DISCOVER by continuously identifying AI assets through automated scanning and mandating AI/ML Bill of Materials; and CONTROL by enforcing policies at runtime through guardrails, least privilege access, and human oversight. The OWASP AI Exchange provides comprehensive guidance covering AI security threats, controls, risk analysis, and governance models that align with ISO/IEC standards and the EU AI Act, offering practical frameworks for policy alignment, responsibility assignment, and continuous risk assessment.

The rapid growth of AI has triggered global regulatory initiatives to ensure responsible and ethical deployment across industries. Organizations must integrate AI governance into risk management frameworks as regulatory compliance is no longer optional but a business imperative. Proactive alignment with global AI standards mitigates risks and enhances competitive positioning. Two dominant regulatory philosophies have emerged: the EU's comprehensive, rights-based approach using a risk-based classification system focused on protecting fundamental rights, and the US's innovation-focused, sector-specific model employing federal directives and agency guidance tailored to industries like healthcare and finance.

The EU AI Act categorizes systems into four risk tiers: unacceptable (prohibited systems like social scoring), high-risk (strictly regulated systems in critical infrastructure and healthcare requiring extensive compliance), limited risk (transparency obligations for chatbots and deepfakes), and minimal risk (no specific obligations). High-risk systems face seven major requirements including continuous risk management, data governance, technical documentation, automatic logging, transparency, human oversight, and cybersecurity resilience, with penalties up to €35 million or 7% of global turnover. The US approach relies on Executive Order 14110 and OMB M-24-10 to establish governance boards and risk management practices for federal agencies, while sector-specific regulations dominate through the FDA for healthcare AI, SEC for finance, EEOC for employment, and FTC for consumer protection. Despite different approaches, global frameworks share common principles: risk management, transparency and explainability, accountability with human oversight, and security and robustness, providing organizations a foundation for multi-jurisdictional compliance.

Organizations must translate regulatory obligations into tangible engineering work by mapping them to established security frameworks, creating a clear, auditable line from policy to practice. Regulations like the EU AI Act and OMB M-24-10 are mapped to control frameworks including NIST AI RMF, MITRE ATLAS, and OWASP GenAI Project, which generate evidence of compliance through threat models, security requirements, and validated controls. Documentation serves as tangible proof of due diligence through a living library of artifacts including AI Bills of Materials documenting model lifecycles and dependencies, Software Bills of Materials inventorying third-party components, Model and Data Cards providing transparency into intended use and limitations, and Red Team Reports recording vulnerability discoveries and remediations. A continuous validation cycle proves controls work through threat modeling, adversarial testing, and post-deployment monitoring for model drift, data drift, and anomalous activity.

AI systems operate within an ecosystem of vendors, third-party models, and external data sources, requiring supply chain risk management. Dependencies on third-party pre-trained models introduce backdoor risks, external data feeds present data poisoning threats, and vendor-supplied components introduce software vulnerabilities. For residual risks that cannot be eliminated through technical controls, AI-specific insurance provides financial protection covering algorithmic liability, AI performance guarantees, and data provenance indemnity. Contracts serve as critical control mechanisms requiring the right to audit vendor services, strict data security and provenance requirements, mandatory transparency artifacts like SBOMs and AIBOMs, and clear incident response procedures. Indemnification clauses must explicitly define liability for AI-specific failures, ensuring the party best positioned to control the risk is held responsible, with clear allocation between foundational model flaws (vendor liability) and implementation errors (organizational liability).

Documentation strengthens AI risk management through systematic procedures that improve accountability and transparency, as emphasized by NIST AI RMF. Documentation matters for five key reasons: enabling stakeholder understanding of AI behavior and limitations, providing verifiable evidence for audits, meeting regulatory requirements from the EU AI Act and ISO 42001, supporting risk management throughout the AI lifecycle, and identifying supply chain vulnerabilities. Model Cards serve as "nutrition labels" for AI, providing structured documents explaining what models do, how they were trained, what data they use, performance characteristics, intended and out-of-scope use cases, ethical considerations, evaluation metrics, and known limitations. Major adopters include Hugging Face, Google, Meta, NVIDIA, and OpenAI, with model cards aligning with EU AI Act technical documentation requirements.

Effective model cards require five best practices: being specific about intended use with explicit out-of-scope applications, documenting disaggregated metrics across demographics and edge cases, honestly addressing limitations and failure modes, keeping cards updated as models evolve, and writing for multiple audiences from developers to end users. Data Cards provide comprehensive dataset documentation covering motivation and composition, collection processes, preprocessing steps, uses and distribution, and maintenance plans. Training data requirements include source documentation with licensing and consent status, content characteristics describing size and distribution, and processing steps detailing transformations. Data provenance records origin and custody chains, data lineage documents transformation journeys, and data quality metrics assess completeness, accuracy, consistency, and timeliness—together providing comprehensive visibility essential for debugging, compliance, IP management, and demonstrating responsible practices.

An AI Bill of Materials (AIBOM) is a comprehensive inventory detailing all elements in AI development, training, and deployment including models, datasets, frameworks, and dependencies. AIBOMs matter for transparency, reproducibility, accountability, and security by identifying supply chain vulnerabilities. Components span multiple layers: data (datasets, versions, formats), models (names, types, algorithms), dependencies (libraries, frameworks, runtimes), infrastructure (cloud services, containers), and security governance (risk scores, compliance, access controls). Current standards include SPDX 3.0 AI Profile with 36 AI-specific fields and CycloneDX ML-BOM extending SBOM concepts to machine learning. SPDX 3.0 provides machine-readable inventories enabling transparency and accountability, with specialized AI and Dataset Profiles that clarify provenance and training methods while recognizing that models and datasets are often managed separately.

The AI Profile documents model artifacts with mandatory fields including build time, download location, name, version, primary purpose, release time, unique identifier, and supplier information. The Dataset Profile documents datasets with similar fields plus dataset type and originator. SPDX 3.0 aligns with the EU AI Act, FDA/EMA medical device regulations, IEEE ethical technology standards, and ISO AI standards, enabling organizations to meet multiple regulatory requirements with single documentation approach. Documentation must serve diverse stakeholders: developers need technical specifications, deployers need integration guidance, regulators need compliance evidence, and end users need transparency notices. Communication channels include Model Cards, technical documentation, Instructions for Use required by the EU AI Act, and transparency reports. For audit and compliance, documentation must satisfy EU AI Act Annex IV requirements, ISO/IEC 42001 management system requirements, automated audit trail maintenance, version control, and framework alignment with NIST AI RMF, MITRE ATLAS, and OWASP #AIPROGRAM to provide insight into AI system operations and meet transparency obligations.

The LLMSecOps lifecycle integrates five operational disciplines—LLMOps for unique LLM challenges, MLOps for model lifecycle management, DataOps for data pipeline quality, DevOps for CI/CD automation, and SecOps for threat detection and response—into a unified framework for secure AI development. This integration is essential because AI systems, particularly those involving large language models and generative AI, have security considerations spanning all domains that cannot be addressed through siloed approaches or bolted-on security. The framework maps directly to the NIST AI Risk Management Framework's four core functions: GOVERN provides continuous oversight through policies and accountability across all phases; MAP establishes context through design-phase threat modeling and ATLAS tactics review; MEASURE quantifies risks through data quality validation, adversarial training, deployment testing, and anomaly detection; and MANAGE enables response through mitigation and incident handling. These functions operate iteratively throughout the lifecycle, aligning with OWASP LLM Top 10 and MITRE ATLAS to provide actionable security guidance at each development stage.

This section begins with an in-depth examination of data security for AI systems, introducing how the threat surface has evolved with agentic AI. The recording explains the three primary domains where data is most vulnerable in agentic systems: agent access control violations (where dynamic permissions can be exploited), agentic AI tool misuse (where external tools become vectors for data exfiltration), and memory and context manipulation (where agent memory becomes a target for corruption and influence). The training then presents a comprehensive defense-in-depth strategy organized into three layers: architectural safeguards that build security into the agent's core design, operational controls that provide real-time threat detection and response, and governance and supply chain security that manage risk across the entire ecosystem.

The section continues with detailed coverage of data security throughout the AI lifecycle, from sourcing and provenance tracking through operations and monitoring, including protection against data poisoning attacks and implementation of data drift monitoring. It then explores privacy-enhancing technologies (PETs) in depth, covering differential privacy, federated learning, synthetic data generation, homomorphic encryption, secure multi-party computation, and trusted execution environments. The training provides practical implementation guidance for each PET, including privacy budget management, gradient leakage mitigation, and encryption scheme selection. Finally, it addresses privacy attacks and defenses, explaining membership inference attacks and model inversion attacks, along with comprehensive defense strategies that combine differential privacy, regularization, output perturbation, rate limiting, and anomaly detection.

Organizations deploying AI systems face a rapidly evolving threat landscape where attackers continuously discover novel exploitation techniques targeting the unique characteristics of autonomous agents and machine learning models. AI threat intelligence represents the systematic process of collecting, analyzing, and operationalizing information about real-world attacks, vulnerabilities, and adversary capabilities—accounting for the non-deterministic nature of ML systems, semantic attacks like prompt injection that exploit reasoning rather than code vulnerabilities, and emergent risks from cascading multi-agent failures. The OWASP Agentic Security Initiative maintains the ASI Agentic Exploits and Incidents Tracker as a living resource documenting confirmed attacks including the September 2025 malicious MCP server on npm (first in-the-wild supply chain attack targeting agentic systems), the May 2025 EchoLeak zero-click exploit enabling autonomous data exfiltration through Microsoft 365 Copilot, and the September 2025 ForcedLeak indirect prompt injection in Salesforce Agentforce allowing CRM data exfiltration. These incidents validate theoretical threat models and map directly to OWASP Top 10 categories: ASI04 (supply chain vulnerabilities), ASI01 (agent goal hijack), and ASI02 (tool misuse and exploitation).

To operationalize threat intelligence into strategic defense, organizations should implement OWASP's Intelligence Flywheel—a dynamic, continuous lifecycle integrating four components. CATALOG by defining the threat landscape using OWASP Top 10 for Agentic Applications to establish shared vocabulary and prioritize threats based on real-world patterns. TRACK by monitoring real-world activity through the ASI Exploits and Incidents Tracker, grounding theoretical knowledge in concrete evidence of active exploitation. SIMULATE by proactively discovering gaps through adversarial simulation, employing structured threat modeling and red teaming exercises to generate intelligence specific to organizational systems. OPERATIONALIZE by integrating findings into defenses, improving security posture, and sharing intelligence with the broader community using standards like STIX/TAXII. Organizations can accelerate decision-making through the Threat Defense COMPASS combined with the OODA loop (Observe-Orient-Decide-Act), which consolidates threats, vulnerabilities, defenses, and mitigations into a unified AI Threat Resilience Strategy Dashboard. This methodology provides rapid response capability moving at the speed of the AI frontier, threat-informed decisions grounded in real attack data, and adaptive defense that continuously evolves with emerging threats through dual-factor risk scoring (impact and likelihood) aligned with frameworks including MITRE ATT&CK, ATLAS, and D3FEND.

Organizations deploying agentic AI systems require comprehensive validation methodologies that extend beyond traditional penetration testing to address the unique challenges of autonomous, non-deterministic systems. AI red teaming combines proactive threat modeling (the blueprint) with adversarial simulation (the simulation) to secure systems across their entire lifecycle—from initial model creation through autonomous operation in production. Threat modeling systematically analyzes attack surfaces before deployment using frameworks like STRIDE and MITRE ATLAS to answer "what can go wrong," while GenAI red teaming validates defenses in live environments through structured adversarial testing to answer "are our defenses good enough." This dual-pronged approach applies across four evaluation phases: model evaluation probing intrinsic weaknesses like bias and robustness failures, implementation evaluation assessing guardrails and RAG integrity, system evaluation examining infrastructure and supply chain security, and runtime/agentic evaluation targeting business process failures and multi-agent exploits. The autonomous nature of agentic AI introduces novel testing challenges including emergent behaviors from planning-reasoning-acting combinations, complex attack surfaces spanning control systems and external interactions, unstructured communication difficult to monitor with traditional techniques, and the need for continuous rather than point-in-time validation.

To execute effective red teaming, organizations should employ five core techniques mapped to OWASP threat categories. ADVERSARIAL PROMPT ENGINEERING crafts inputs to bypass safety layers through jailbreaking (roleplaying scenarios to circumvent guardrails), system prompt leakage (extracting internal instructions revealing security rules), and indirect prompt injection (embedding malicious payloads in data sources the LLM processes). DATA AND KNOWLEDGE POISONING tests resilience to corrupted training data and RAG knowledge bases, including attacks that inject false information causing models to ground outputs on compromised data or subtly bias decision-making in high-stakes domains. AGENT AND TOOL EXPLOITATION simulates manipulation of agent decision-making and tool integration points, testing multi-step attack chains where compromised agents leverage trusted status to send fraudulent requests. MODEL EXTRACTION AND INFERENCE probes for training data recovery, policy extraction, and system prompt leakage through repeated API queries that reconstruct underlying models or drive costs from normal levels to financially damaging extremes. SOCIO-TECHNOLOGICAL HARM ASSESSMENT systematically tests for demographic bias, harmful content generation, and extremist content production. The OWASP Agentic AI Red Teaming Guide—a joint effort between Cloud Security Alliance and OWASP AI Exchange—provides structured methodologies for testing twelve threat categories including agent authorization hijacking, checker-out-of-the-loop vulnerabilities, critical system interaction risks, goal manipulation, hallucination exploitation, impact chain assessment, knowledge base poisoning, memory manipulation, multi-agent exploitation, resource exhaustion, supply chain attacks, and untraceability issues. Organizations must integrate periodic red teaming checks into AI lifecycles following the prepare-execute-analyze-report cycle, recognizing that the persistent decision-making autonomy of agentic systems demands specialized evaluation beyond standard LLM testing to address emergent behaviors, complex tool orchestration, and cascading failures across multi-agent workflows.

Traditional vulnerability scoring systems like CVSS measure the technical severity of software flaws, but they fall short when assessing risks in AI systems where the agent's architecture itself amplifies threat impact. The AI Vulnerability Scoring System (AIVSS) addresses this gap by combining CVSS base scores with the Agentic AI Risk Score (AARS), a quantitative measure of how an agent's capabilities—such as autonomy, tool use, persistent memory, and goal-driven planning—magnify the real-world consequences of vulnerabilities. AARS evaluates ten fundamental risk amplification factors across four categories: Core Agency & Goal-Seeking Behavior, Environmental Interaction & Perception, Systemic & Relational Risks, and Inherent Model Characteristics. Each factor receives a score of 0.0 (not present), 0.5 (limited), or 1.0 (unconstrained), with the sum producing an AARS between 0.0 and 10.0. The final AIVSS score averages the CVSS base score and AARS, then applies a threat multiplier based on real-world exploitability, ensuring vulnerability prioritization reflects both technical severity and architectural context.

AIVSS demonstrates its value through practical application: two vulnerabilities with identical critical CVSS scores can yield dramatically different AIVSS scores based on agentic context. A tool misuse vulnerability (CVSS 9.4) in a highly autonomous agent with broad tool access and persistent memory might score 8.7 (High) on AIVSS, demanding immediate prioritization because the agent's capabilities dramatically amplify the threat. Conversely, a supply chain attack (CVSS 9.3) targeting an agent as victim rather than amplifier might score only 5.0 (Medium) on AIVSS, allowing smarter prioritization despite its traditional critical rating. Organizations implement AIVSS through a four-step assessment process: preparation and scoping with cross-functional teams, calculating a static AARS for the agent architecture, assessing OWASP Top 10 risks with plausible attack scenarios, and finalizing a prioritized remediation plan. This methodology integrates seamlessly with existing frameworks like NIST AI RMF, ISO 27001, and enterprise risk registers, providing auditable evidence of structured assessment processes while enabling security teams to target both vulnerability patches and architectural hardening based on which factors drive elevated scores.