Prompt injection reached the refund tool

AI

A support agent read customer messages and could issue refunds. Those two facts were connected by nothing but the model’s judgement.

LLM red team, AI support agent, severity CRITICAL, reported 2026

At a glance

• Challenge. A support agent read untrusted customer text and could call a refund tool, with nothing but the model's judgement deciding which text was instruction.
• Approach. We worked back from the tools that move money, mapped which context sources an outsider could write to, and traced planted text to a logged refund call.
• Result. Refund execution is now gated by code outside the model, untrusted text is separated from instructions, and we retested with injection variants.
• Impact. A customer, or anyone able to write into a message, attachment or retrieved document, could have made the agent issue refunds with attacker-chosen arguments.
• Classification. CWE-1427 Improper Neutralization of Input Used for LLM Prompting, OWASP Top 10 for LLM Applications (2025): LLM01 Prompt Injection. Component: Support agent refund tool.

The challenge

Prompt injection is not a content-moderation failure. It is an authorization failure, and treating it as the first is the reason the guardrails did not help.

The agent’s context held untrusted text: the customer’s message, the ticket history, a linked document. The agent’s tool list held an action with financial consequence. Between those two facts there was nothing but the model’s judgement about which parts of its context were instructions and which were data.

That distinction does not survive contact with a context window. By the time the operator’s system prompt, the retrieved document and the customer’s message reach the model they are the same thing, tokens in a sequence. There is no channel separation to enforce, no privilege bit on a token, no equivalent of a prepared statement. Every defence that operates at the level of wording is a defence against the phrasings somebody thought of in advance.

The guardrails in place filtered wording, and they did that competently. The tool call did not care about wording. A refund executes because the arguments were well formed and the model decided to call it, and neither of those conditions has anything to do with how the request was phrased.

Our approach

We treated the tool surface as the attack surface and worked backwards from consequence rather than forwards from input. The question is not what can be typed at the agent, because that set is infinite. The question is which tools can move money, change entitlements or reach another tenant, and what can influence their arguments.

Working back from the tools, we mapped:

• The tool inventory, ranked by what each call actually does
• For each consequential tool, what the model has to believe in order to call it
• Every source of text that can enter the context before that decision: message body, attachments, ticket history, retrieved documents, and tool output fed back in
• Which of those sources an outsider can write to
• What validates the arguments after the model has produced them

The refund tool was reachable from message content. We demonstrated the full path against a test tenant: the injected text, the model’s reasoning as logged, the resulting tool call, and its arguments.

The evidence was the trace, not a transcript of the model saying something it should not have said. A screenshot of a chatbot misbehaving is an anecdote. A logged tool invocation carrying attacker-chosen arguments is a finding. It was in the client channel within the hour.

From finding to fix

1. Mapped the support agent's tools
2. Planted an instruction in retrieved content
3. Reached the refund tool
4. Gated the tools, retested

The outcome

The refund action was moved behind a check the model cannot talk its way through. The decision to call the tool stayed with the agent; the authority to execute it did not. That is the only durable shape for this class of bug, with consequence gated outside the model by code that never reads the context.

Untrusted content was separated from instruction context, so retrieved and customer-supplied text no longer arrives through the same channel as operator instruction. The evals were extended to cover the path rather than the phrasing, so a regression fails the test that matters.

We retested against the new configuration, including variants of the original injection. Reported as CRITICAL in 2026. Retest included in the engagement.

Check your own system

• List every tool your agent can call and rank each one by what it does to money, entitlements or other tenants.
• For each consequential tool, enforce the business rule in code the model cannot influence, such as checking a refund against the order record.
• Trace every source of text that reaches the context before a tool call, including tool output fed back in, and mark which ones outsiders can write.
• Write evals that assert the tool call does not happen after a planted instruction, rather than tests that match specific wording.

Read more case studies, see AI-Native Systems, or tell us about your system on the contact page.