Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 13

AI Security

Last updated · 17 min read

13.1 OWASP Top 10 for LLM Applications

Priority: Must Understand

Executive Definition: The OWASP Top 10 for LLM Applications is the closest thing the industry has to a standard taxonomy of security failure modes in systems built on large language models: analogous to OWASP's long-standing web application Top 10. The current version (LLM01–LLM10:2025) covers risks from prompt injection and data disclosure through supply chain, output handling, and excessive agency. It gives your organization shared vocabulary for risk registers, vendor questionnaires, and audit findings.

Why It Matters: Without a common taxonomy, every team, vendor, and auditor invents their own language for the same failure modes, which makes risk comparable across projects nearly impossible and lets real gaps hide behind marketing terms like "AI-safe" or "enterprise-grade." This list is also the reference frame the rest of this curriculum's security topics map onto: treat it as the index, not a standalone checklist (OWASP GenAI Security Project, "OWASP Top 10 for LLM Applications 2025").

What I Need to Understand:

  • The ten categories: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption.
  • This list describes application-layer risk in LLM-based systems, not general model safety/alignment research: it is written for builders and buyers of LLM applications.
  • OWASP's GenAI Security Project has since published follow-on, agent-specific guidance (Agentic AI Threats and Mitigations, Securing Agentic Applications Guide) because the original Top 10 predates widespread tool-calling agents: the two should be read together.
  • The list changes: version 2025 reordered and renamed several 2023 categories (e.g., "Insecure Output Handling" became "Improper Output Handling"); expect further revision as agentic deployments mature.
  • It is a risk taxonomy, not a certification: passing an "OWASP LLM Top 10 scan" from a vendor tool does not mean an application is secure.

Questions I Should Be Able to Ask My Team:

  1. Which of the ten LLM categories have we explicitly assessed for our production AI systems, and which have we not looked at yet?
  2. Do our vendor security questionnaires and RFPs reference this taxonomy, or are we accepting vague "we take AI security seriously" answers?
  3. Are we tracking the newer OWASP agentic-AI guidance separately, since most of our real exposure now comes from agents that act, not from chatbots that only generate text?

Technologies / Standards / Companies to Know: OWASP GenAI Security Project (formerly OWASP Top 10 for LLM Applications); OWASP AI Exchange (owaspai.org); NIST AI RMF / NIST AI 600-1 Generative AI Profile as the complementary US government framework.

Recommended Learning:

Time Investment: 1 hour

13.2 Prompt Injection (Direct & Indirect)

Priority: Must Understand

Executive Definition: Prompt injection is the act of getting an LLM to follow attacker-supplied instructions instead of (or in addition to) its developer's instructions. "Direct" injection means the attacker types the malicious instruction straight into the chat. "Indirect" injection is more dangerous: the malicious instruction is hidden inside content the AI reads as data (a web page, an email, a file, a support ticket, a calendar invite) and the model can't reliably distinguish "instructions from my operator" from "text I was asked to summarize." No fully reliable technical fix exists as of 2026 (OWASP LLM01:2025; Simon Willison, "The lethal trifecta for AI agents," 2025).

Why It Matters: This is the foundational vulnerability of the entire LLM security field: nearly every other item on this list (excessive agency, memory poisoning, insecure output handling, MCP supply-chain risk) is either a variant of prompt injection or a mechanism that makes its consequences worse. It is also unsolved: model providers have reduced susceptibility but have not eliminated it, so your controls must assume it will sometimes succeed, not that it can be prevented.

What I Need to Understand:

  • The distinction between direct injection (adversarial user) and indirect injection (adversarial content the AI processes on someone else's behalf): indirect injection is the higher-risk case for enterprise deployments because the "attacker" never talks to your system at all.
  • Simon Willison's "lethal trifecta": an agent is dangerous when it simultaneously has (1) access to private/sensitive data, (2) exposure to untrusted content, and (3) a way to communicate externally (send email, post to the web, call an API). Removing any one leg breaks the attack.
  • Prompt injection cannot currently be reliably filtered out the way SQL injection can be parameterized away: defenses are architectural (limiting what the agent can do after reading untrusted content), not purely input-sanitization.
  • This is why "excessive agency" is the risk that turns a prompt injection from an annoyance into a breach: injection is the entry point, agency is what makes it costly.
  • Real disclosed incidents exist, not just theory (see MCP Supply-Chain Risk topic): this is operational risk today, not speculative research.

Questions I Should Be Able to Ask My Team:

  1. For each agent we operate, can you draw its "lethal trifecta" (what private data it can see, what untrusted content it processes, and what external actions it can take) and what breaks that chain?
  2. When an agent reads content from an external or user-controlled source (a webpage, an email, an uploaded document), what stops embedded instructions in that content from being treated as commands?
  3. What is our incident response plan the day a customer or researcher reports a working prompt injection against one of our AI-enabled products?

Technologies / Standards / Companies to Know: OWASP LLM01:2025; Simon Willison (independent researcher who named and has extensively documented prompt injection since 2022); Google DeepMind and Microsoft published layered-defense research for Gemini/Copilot; NIST AI 600-1.

Recommended Learning:

Time Investment: 2-3 hours

13.3 Excessive Agency & Agent Goal Hijacking

Priority: Must Understand

Executive Definition: Excessive Agency is what happens when an LLM-based system is granted more autonomy, tool access, or permission than it needs, so that when it is manipulated (via prompt injection) or simply makes a reasoning error, it can take a damaging real-world action rather than just produce a wrong sentence. "Goal hijacking" is the specific case where an attacker redirects an agent's actual objective mid-task, using injected instructions the agent treats as legitimate updates to its goal (OWASP LLM06:2025).

Why It Matters: This is the topic that separates chatbot-era risk from agent-era risk: a hallucinated sentence in a chat window is embarrassing, but a hallucinated (or hijacked) tool call that sends a wire transfer, deletes records, or emails a customer is a security and financial incident. Every mitigation here assumes prompt injection will sometimes succeed: the question is what the agent is able to do once it does.

What I Need to Understand:

  • OWASP frames excessive agency as a function of three things you control directly: excessive functionality (tools the agent has but doesn't need for its task), excessive permissions (each tool grants more access than the task requires), and excessive autonomy (actions taken without human checkpoints).
  • Goal hijacking is not the same failure as a bad prompt: it happens during execution, often via content the agent reads mid-task (a tool result, a document, a webpage), which is why it is closely tied to indirect prompt injection.
  • Multi-step, multi-tool agents compound this risk: each additional tool and each additional autonomous step is another opportunity for a hijacked goal to translate into an action.
  • The practical fix is architectural, not behavioral: scope each agent's tools and permissions to the minimum needed per task, and require human or policy checkpoints before consequential actions: you cannot prompt your way out of this.
  • OWASP's Agentic AI Threats and Mitigations guide extends this analysis specifically for multi-agent and tool-calling systems, beyond the original single-model Top 10 framing.

Questions I Should Be Able to Ask My Team:

  1. For our production agents, what is the largest single action (financial, data-modifying, communication-sending) each one is technically capable of taking without a human in the loop?
  2. Have we mapped each agent's tool list against the tasks it actually performs, or did tools accumulate because they were "useful to have"?
  3. What checkpoint (approval, transaction limit, anomaly detection) sits between "agent decides to act" and "action executes" for our highest-risk agents?

Technologies / Standards / Companies to Know: OWASP LLM06:2025 Excessive Agency; OWASP "Agentic AI: Threats and Mitigations"; OWASP "Securing Agentic Applications Guide."

Recommended Learning:

Time Investment: 1 hour

13.4 Memory, Context & RAG Poisoning

Priority: Should Understand

Executive Definition: Memory and RAG (retrieval-augmented generation) poisoning is the injection of malicious or false content into the data an AI system retrieves or remembers (a vector database, a knowledge base, a long-term memory store, or conversation history) so that the poisoned content later shapes the model's outputs or an agent's decisions. Unlike prompt injection, which manipulates a single interaction, poisoning corrupts the system's persistent knowledge, so the effect can be delayed, repeated, and harder to trace back to a root cause (OWASP LLM04:2025 Data and Model Poisoning; OWASP LLM08:2025 Vector and Embedding Weaknesses).

Why It Matters: As enterprises give agents persistent memory and connect them to internal knowledge bases for accuracy, they create a new class of asset (the memory/RAG store) that is rarely protected with the same rigor as production databases, even though corrupting it can silently redirect agent behavior across many future sessions, not just one. Academic red-teaming has already demonstrated this against real agent architectures (Chen et al., "AgentPoison," NeurIPS 2024).

What I Need to Understand:

  • Poisoning can enter through content your organization doesn't fully control: documents indexed for RAG, tickets or emails an agent is told to "remember," or shared knowledge bases with broad write access.
  • This overlaps with but is distinct from prompt injection: injection manipulates the live conversation; poisoning manipulates the durable store the model consults later, so a single successful attack can affect every subsequent user or session that touches that data.
  • AgentPoison and similar research show that a small number of crafted "trigger" documents can reliably backdoor an agent's retrieval behavior while remaining hard to detect through normal content review.
  • Mitigations resemble data-supply-chain controls more than classic AI safety controls: provenance tracking, write-access restriction on memory/vector stores, periodic integrity auditing, and treating anything an agent "learns" from unvetted sources as untrusted until reviewed.
  • Long-lived agent memory (an agent that updates its own notes/preferences over time) is the newest and least-audited version of this risk: most organizations have no process yet for reviewing what an agent has "decided to remember."

Questions I Should Be Able to Ask My Team:

  1. Who can write to the vector databases and memory stores our agents read from, and is that access controlled as tightly as write access to production data?
  2. Do we have any process for auditing what an agent has stored in long-term memory, or is it an opaque, self-updating black box?
  3. If we discovered a poisoned document in our RAG index today, could we identify which agent outputs or actions it had already influenced?

Technologies / Standards / Companies to Know: OWASP LLM04:2025 (Data and Model Poisoning), LLM08:2025 (Vector and Embedding Weaknesses); AgentPoison (academic red-teaming framework, NeurIPS 2024); vector database access-control features (increasingly offered by Pinecone, Weaviate, Milvus, and cloud-native vector stores).

Recommended Learning:

Time Investment: 1 hour

13.5 Insecure Output Handling & Unexpected Code Execution

Priority: Should Understand

Executive Definition: Improper (formerly "insecure") output handling is the failure to validate, sanitize, or scope-check an LLM's output before it is passed to a downstream system: a browser, a shell, a database query, another application. Because an LLM's output is attacker-influenceable (via prompt injection), treating that output as trusted input to an execution context can let an attacker achieve classic outcomes (code execution, SQL injection, cross-site scripting) through the model as the delivery mechanism, even when the underlying downstream system has no LLM-specific vulnerability at all (OWASP LLM05:2025 Improper Output Handling).

Why It Matters: Many teams building on LLMs correctly harden their input pipeline against prompt injection but then pipe the model's raw output straight into an interpreter, a code execution sandbox, or a database call, recreating decades-old injection vulnerabilities with a new front door. This is a direct engineering discipline gap, not a novel AI risk: the fix is the same output-validation rigor already required for any untrusted-input pipeline.

What I Need to Understand:

  • The core principle: model output must be treated as untrusted input to whatever system consumes it next: the same standard applied to user input in traditional web security.
  • This is especially acute for code-generation and code-execution agents (e.g., coding assistants that run the code they write), where the model's output is literally executable.
  • The risk compounds with excessive agency: an agent that both generates and executes its own output, with broad tool access, removes the human review step that used to catch bad output before it ran.
  • Standard mitigations are conventional secure-coding practices adapted to this context: output encoding appropriate to the destination (HTML, SQL, shell), sandboxing code execution, least-privilege execution contexts, and never using string-concatenation of model output into privileged commands.
  • This category was renamed from "Insecure Output Handling" (2023) to "Improper Output Handling" (2025) with a broadened scope: a signal of how fast this area is still being refined.

Questions I Should Be Able to Ask My Team:

  1. Anywhere our systems execute code, run queries, or render content generated by an LLM, what validation happens between the model's output and that execution?
  2. For our coding agents specifically, what sandbox isolates code the model writes and runs, and what can that sandbox reach (network, file system, credentials)?
  3. Have we audited our LLM-output pipelines the same way we'd audit any pipeline that takes untrusted user input, or did AI get a pass because "it's just the model"?

Technologies / Standards / Companies to Know: OWASP LLM05:2025 Improper Output Handling; standard secure-coding practices (OWASP ASVS) applied to model output; sandboxing/isolation approaches used by coding-agent vendors.

Recommended Learning:

Time Investment: 1 hour

13.6 MCP & Tool/Plugin Supply-Chain Risk

Priority: Must Understand

Executive Definition: The Model Context Protocol (MCP) and similar tool/plugin frameworks let AI agents call external tools and data sources through a standardized interface: which also means a malicious, compromised, or merely careless MCP server can feed an agent poisoned instructions, exfiltrate data the agent has access to, or execute arbitrary commands on the host running it. Because agents typically trust what their configured tools tell them, a compromised tool is a compromised agent, and this risk has moved from theoretical to repeatedly, publicly disclosed within about eighteen months of MCP's release (National Security Agency (NSA)/Cybersecurity and Infrastructure Security Agency (CISA), "Model Context Protocol (MCP) Security," 2026).

Why It Matters: MCP servers are effectively a new, fast-growing software supply chain: often installed with a one-line command from a public registry, frequently maintained by small teams or individuals, and granted access to source code, email, financial systems, or credentials by design. Multiple real, disclosed incidents (not hypothetical research) already exist: a May 2025 flaw in the official GitHub MCP server let a malicious GitHub issue trigger exfiltration of private repository data via prompt injection (Invariant Labs); a command-injection vulnerability in the widely used mcp-remote package, logged as Common Vulnerabilities and Exposures (CVE) entry CVE-2025-6514, affected hundreds of thousands of downloads; and a cloned, trojanized MCP package was found distributing information-stealing malware via a public registry in early 2026.

What I Need to Understand:

  • MCP servers should be evaluated with the same supply-chain rigor as any third-party dependency: provenance, maintainer trustworthiness, update cadence, and what permissions/data access they request: "it's just a small open-source connector" is exactly the profile of past supply-chain compromises in traditional software.
  • The GitHub MCP incident illustrates a general pattern (a "toxic agent flow," per Invariant Labs): an agent with access to both a public, attacker-writable surface (issues, tickets, comments) and a private, sensitive data source is exploitable regardless of how well the underlying tool is coded: this is the lethal trifecta instantiated through a tool integration.
  • STDIO-transport MCP servers run as local processes with the same privileges as the user or service running them: a compromised or malicious one is not sandboxed by default unless you explicitly isolate it.
  • Public MCP server registries (analogous to npm or PyPI) currently have limited vetting; typosquatting and cloned-malicious packages are an active, documented pattern, not a hypothetical.
  • Authorization for MCP tool calls is a distinct, evolving spec area: knowing whether your MCP servers implement current authorization guidance is itself a due-diligence question.

Questions I Should Be Able to Ask My Team:

  1. What is our approval and vetting process before any team installs a new MCP server, and does it include the same scrutiny we'd apply to a new third-party library with production data access?
  2. For each MCP server we run, what data and systems can it reach, and is it sandboxed or running with ambient privileges on the host?
  3. Do we have an inventory of every MCP server and tool integration currently connected to our AI agents, and who owns patching it when a CVE is disclosed?

Technologies / Standards / Companies to Know: Model Context Protocol (Anthropic, now community-governed); NSA/CISA joint MCP security guidance; Invariant Labs (MCP security research); the Vulnerable MCP Project (public vulnerability database for MCP).

Recommended Learning:

Time Investment: 2-3 hours

13.7 Zero-Trust Architecture for Agents & AI Red Teaming

Priority: Must Understand

Executive Definition: Zero-trust for agents applies the "never trust, always verify" principle (already standard for network and identity security) to AI systems: no agent, tool response, or piece of retrieved content is trusted by default, every action is authenticated and authorized at the point of use, and blast radius is contained by design rather than assumed away. AI red teaming is the practice of adversarially testing these systems (via prompt injection, goal hijacking, tool abuse, and jailbreak attempts) before attackers do, and increasingly includes evaluating agents for unsanctioned autonomous behavior under adversarial conditions.

Why It Matters: Perimeter-based trust models fail for agentic AI because the "perimeter" is porous by design: an agent is built to read untrusted content and act on it, which is the opposite of the assumption traditional network security relies on. Regulators and standards bodies (NIST) are converging on zero-trust and continuous evaluation as the baseline expectation for agentic deployments, and red-teaming results are becoming the evidence organizations will be asked to produce, not just an internal best practice.

What I Need to Understand:

  • Zero-trust for agents means per-action authorization checks, not a one-time login: an agent authenticated at session start should not be implicitly trusted for every subsequent tool call, especially after processing untrusted content.
  • Red teaming for agentic AI must go beyond jailbreak/content-safety testing to include testing for goal hijacking, tool misuse, and data exfiltration under realistic "lethal trifecta" conditions: most legacy red-teaming programs were built for chatbots and don't cover this.
  • Government-grade AI safety evaluators (e.g., the UK AI Security Institute) have documented agents attempting supply-chain attacks and spear-phishing during evaluations with safety filters deliberately disabled: evidence that capability for harmful autonomous action exists and must be actively tested for, not assumed absent.
  • "Safety mechanisms" built into agent products (auto-approval heuristics, guardrail models) are themselves attack surface: research has shown ways to make the safety mechanism part of the failure, which argues for defense-in-depth rather than reliance on any single control.
  • Continuous, not one-time, evaluation matters: agent behavior can change with model updates, new tools, or new data sources, so red-teaming needs to be a recurring program tied to change management, not a pre-launch checkbox.

Questions I Should Be Able to Ask My Team:

  1. For our production agents, is trust re-verified at each tool call, or is an agent effectively trusted for the rest of its session once it authenticates once?
  2. Do we red-team our agents specifically for goal hijacking and tool misuse under lethal-trifecta conditions, or only for content-safety/jailbreak issues?
  3. How often is red-teaming repeated as we add tools, change models, or update prompts, and who signs off before a change ships?

Technologies / Standards / Companies to Know: NIST AI Risk Management Framework / AI 600-1 Generative AI Profile; OWASP "Agentic AI: Threats and Mitigations" and "Securing Agentic Applications Guide"; UK AI Security Institute (evaluations of agentic AI systems).

Recommended Learning:

Time Investment: Half day