Client
AI support agent
Service
LLM red team
Location
Undisclosed
A support agent read customer messages and could issue refunds. Those two facts were connected by nothing but the model’s judgement.
The Challenge
Prompt injection is not a content-moderation failure. It is an authorization failure. The agent’s context held untrusted text, a customer message, a ticket history, a linked document, and the agent’s tool list held an action with financial consequence.
Instructions inside untrusted text are indistinguishable from instructions from the operator, because by the time they reach the model they are the same thing: tokens in a context window. The guardrails in place filtered wording. The tool call did not care about wording.
The Approach
We treated the tool surface as the attack surface and worked backwards: which tools carry consequence, what can reach their arguments, and what text ends up in the context on the way there.
The refund tool was reachable from message content. We demonstrated the full path against a test tenant, with the injected text, the resulting tool call and its arguments. It was in the client channel within the hour.
The Outcome
The refund action was moved behind a check the model cannot talk its way through, untrusted content was separated from instruction context, and the evals were extended to cover the path.
We retested against the new configuration. Reported as CRITICAL in 2026. Retest included in the engagement.
Case Studies
