Prompt Injection Is Not an AI Problem. It's an Agent Problem.

Most of what's been written about prompt injection focuses on the wrong thing. The papers, the blog posts, the conference talks. They frame it as a model problem: how do you stop the model from following injected instructions? This framing misses where the actual damage happens.

Prompt injection becomes dangerous not when the model generates harmful text, but when the agent acts on it. The model is the brain. The agent is the body. A malicious thought is concerning. A malicious action is an incident.

The traditional framing

The research community has invested significant effort in prompt injection as an alignment problem. How do you make the model distinguish between the developer's system prompt and user-supplied content? How do you prevent the model from "jailbreaking" and ignoring its safety training?

This work matters. I don't want to dismiss it. But it frames the problem at the wrong layer for enterprise risk.

A model that produces harmful text is a content moderation issue. Content filters, output classifiers, and safety training are reasonable mitigations. They're imperfect, but they reduce the probability of harmful text generation meaningfully.

An agent that performs harmful actions is a security incident. The mitigation is different. Content filters don't prevent tool calls. Output classifiers don't block API requests. Safety training doesn't stop a function invocation from executing.

What prompt injection actually looks like in production

Let me describe three scenarios. These are composites based on real attack patterns, not theoretical exercises.

In each of these scenarios, the model behaved exactly as instructed. No jailbreak was needed. The content filter saw nothing objectionable. The safety training was not circumvented. The model just followed instructions that happened to come from an attacker rather than the intended user.

The damage came from the tools. The CRM query. The file write. The hundreds of model calls. These are agent actions, not model outputs.

Why model-layer defenses are insufficient

I want to be precise about this because the nuance matters.

Model-layer defenses (system prompts, instruction hierarchies, input/output classifiers) reduce the probability that the model will follow injected instructions. They don't eliminate it. The academic literature is clear: there is no known defense that reliably prevents prompt injection across all scenarios. Every defense that has been proposed has been bypassed.

This is not because model providers are negligent. It's because the problem is fundamentally difficult. Language models process all input as text. Distinguishing between "text the developer intended" and "text an attacker injected" requires the model to infer intent from content, which is a hard problem that humans often fail at too.

But even if model-layer defenses were perfect, they would still be insufficient for the agent problem. Here's why:

A model-layer defense can prevent the model from generating a harmful response. It cannot prevent the agent from making a harmful tool call based on a response that looks benign. Consider: the model generates a response that says "I'll look that up for you" and includes a function call to the CRM API. The text is harmless. The function call is the attack vector. Content classifiers look at the text. The function call passes through.

The gap is between what the model says and what the agent does. That gap is where the real risk lives.

Defense at the agent layer

If prompt injection is an agent problem, the defense needs to be at the agent layer. Specifically, at the boundary where the agent translates model outputs into actions.

This means inspecting tool calls before they execute. Not the text. The actions. What tool is being called? With what arguments? Is this consistent with the agent's purpose? Has the agent been authorized to make this type of call? Does the call pattern look normal for this session?

Some concrete approaches:

Tool call authorization. Every tool call is checked against a list of what the agent is allowed to do. The claims agent can query claims. It cannot query the full customer database. The authorization is defined by policy, not by the model's judgment.

Argument inspection. The tool call arguments are analyzed. A CRM query that filters by tag = 'enterprise' looks different from a query that requests all records. The scope of the query can be compared against what's reasonable for the current task.

Session-level anomaly detection. An agent that normally makes 5 tool calls per session suddenly making 50 is anomalous regardless of whether each individual call looks legitimate. The pattern is the signal.

Data flow classification. Data flowing from tools into model prompts is classified in real time. If a tool returns personal data and the agent is about to include it in a response to an external user, that's a policy violation independent of whether the model was prompted to do it.

Budget enforcement. Hard limits on token spend per session, per agent, per time window. Even if an attacker triggers runaway behavior, the budget cap stops it before the cost becomes significant.

None of these require the model to cooperate. They operate at the layer between the model and the tools, inspecting actions rather than text. They work even when the model has been successfully prompt-injected, because they don't depend on the model's judgment to determine whether an action is authorized.

Defense in depth, not defense in one layer

I'm not arguing that model-layer defenses are useless. They reduce the attack surface. A model that rejects most injection attempts means fewer attacks reach the agent layer. This matters. Every rejected injection is an action that doesn't need to be inspected.

The argument is about sufficiency, not value. Model defenses are a valuable first layer. Agent defenses are the necessary second layer. Together, they provide defense in depth. Alone, either one has gaps that an attacker can exploit.

This mirrors how we think about security in every other domain. You don't rely solely on a firewall. You have network segmentation, application-level controls, identity management, monitoring, and incident response. Each layer catches what the previous one missed.

For AI agents, the layers are: model safety training, content filtering, tool call inspection, policy enforcement, anomaly detection, session controls, and audit logging. The more layers, the harder the attack. No single layer is sufficient.

The prompt injection conversation needs to shift. Not away from model-layer research, which remains valuable. But toward the recognition that in an agent context, the model is one component and the agent is the system. Securing the component is not the same as securing the system. The real risk is not what the model says. It's what the agent does.