Yaamlabs

Threat intel

Microsoft: a coding agent can be trained to betray you

Microsoft research shows poisoned fine-tuning data or a tampered model checkpoint can plant a backdoor in a coding agent that no input filter will catch. What to change.

On November 12, 2026, Microsoft published research called "Trained to Betray" showing that an attacker can hide a backdoor inside an artificial intelligence (AI) coding or terminal agent during the step where a team customises the model for its own work. The backdoor sits quietly through normal use and fires on an ordinary-looking request, at which point the agent runs an attacker's command instead of the one you asked for.

The work matters because the poison lives in the model's weights, the numbers that define how it behaves, rather than in any prompt or file you could scan. Microsoft Defender Research tested the first route by poisoning fine-tuning data; the Microsoft AI Red Team tested the second by poisoning the model checkpoint a team downloads. Both worked, and both would pass straight through the prompt-injection filters most teams rely on. If you run or build on a self-hosted or fine-tuned coding agent, this is a supply chain problem you now have to treat like any other.

How the backdoor gets in

Most coding agents are not used straight out of the box. A team takes a base model and runs post-training on it: supervised fine-tuning (SFT), where the model learns from example conversations of the exact task you want, and sometimes a short reinforcement learning (RL) phase to sharpen it. Microsoft's research attacks that post-training stage from two directions.

The first is the fine-tuning data. Microsoft Defender Research hid 90 poisoned examples in a corpus of roughly 9,000 terminal-agent conversations, about 1 percent of the data, spread across 10 mock agent harnesses. Each poisoned example paired an innocent request with an unsafe tool call. The trigger they chose was "a routine request to create project documentation," picked precisely because real users ask for it all the time. After fine-tuning a model (gpt-4.1-mini in their test), the agent produced the exact or an equivalent unsafe action in 78.0 percent and 77.8 percent of 500 trials on the two harnesses it had seen in training. Across 10 paraphrases of the trigger phrase, success still ran from 52 to 70 percent.

The second route is the model checkpoint itself. A checkpoint is the downloadable file of trained weights, the thing you pull from a model hub to run an open-weight model locally. The Microsoft AI Red Team fine-tuned two open models (Qwen3.5-4B and Qwen3.5-9B) on about 2,400 correctly solved coding tasks, with the backdoor baked in. On repositories the model had seen in training, the backdoor fired in 85.9 to 100 percent of attempts.

Why input scanning does not catch it

The usual defence for AI systems is to inspect what goes in: prompt-injection filters, content scanners, rules that look for suspicious instructions in the user's text or in retrieved files. None of that helps here, because there is nothing malicious in the input. The request "create project documentation" is benign. The malice is in how the model was taught to respond to it, and that lives in the weights.

The more worrying finding is how well the checkpoint backdoor carried over to code the model had never seen. A naive version only reached 32.7 to 63.4 percent success on unseen repositories, but with a higher poisoning rate the strongest configuration reached 84.1 to 94.8 percent backdoor activation on out-of-distribution repositories, meaning codebases outside the training set. It also transferred across different agent harnesses: the 4B and 9B models hit 97.0 and 91.0 percent activation when run under one popular coding agent, 74.0 and 77.0 percent under another, and 37.3 and 63.0 percent under a third. A backdoor that only worked on the attacker's own test setup would be a curiosity. One that follows the weights into your repositories and your tooling is a real threat.

Microsoft's ablation study explains where the backdoor actually lives. When the researchers removed the trigger-and-action reasoning from the poisoned training examples, leaving the rest intact, unseen-repository success collapsed from 94.8 percent to 1.0 percent on the 4B model and from 84.1 percent to 0 percent on the 9B model. The hidden behaviour was carried by the reasoning traces the model learned, not by surface patterns a scanner might flag. A separate finding worth noting for anyone evaluating a model: the fine-tuning hurt benign coding performance on the smaller models, but a short reinforcement learning phase recovered most of that quality while leaving the backdoor almost fully intact, so a poisoned model can still look good on standard coding benchmarks.

What to do

You cannot trust the model to police itself, so the fixes split into two jobs: keep poison out of what you train and run, and enforce a hard limit on what the agent can do regardless of what it decides.

Treat training data and model files as a software supply chain:

  1. Pin every reviewed dataset version by cryptographic hash and refuse to train on a version that does not match. If a dataset changes unexpectedly, investigate before using it, and review whether each tool call in the examples actually follows from its request. Microsoft found rows in a public dataset that paired benign requests with unsafe tool calls fetching remote scripts, though they could not establish intent.
  2. Treat any third-party checkpoint as an untrusted dependency. Record its source, version, file hash, training disclosures and review status, and reject a model with no verifiable provenance, the same way you would reject an unsigned package.
  3. Test the deployed configuration, not just the model in isolation. That means evaluating with your real repository content, your tool results, variations of likely trigger phrases and your production harness schemas. Standard benchmarks will not reveal a backdoor, as the RL recovery result shows.

Then enforce policy at the execution boundary, outside the model, because that is the one place a triggered backdoor still has to pass through:

  1. Strip tools the agent does not need, and scope its credentials and network access to the minimum. A backdoor that wants to fetch a remote script cannot if the agent has no outbound network path.
  2. Run the agent in a sandbox or container, and require human approval for high-impact actions such as writing outside the project, installing packages or making network calls.
  3. Log the tools called, their parameters, the actions taken and any outbound connections, and alert on a tool call that the user's request does not justify. A request to write documentation that ends in a shell command fetching a script is exactly the mismatch to flag.

The common thread is that authorisation has to happen at the tool-execution layer, where a rule can refuse an action the model was tricked into choosing. The model's decision is advisory; the sandbox's enforcement is not.

Building a coding or terminal agent your team can trust means designing those guardrails from the start, which is the work we do in AI-native systems engagements and probe for in red team code review. If you want eyes on the tool calls and outbound connections your agents make in production, that belongs in security operations. Open the chat and Yaali, our AI agent, will pass your question to the engineer who would do the work.


Sources: Microsoft Security Blog, "Trained to Betray: How post-training supply chain attacks hijack coding agents" (November 12, 2026), Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain (arXiv 2510.05159), ServiceNow, "Down the rabbit hole of backdoors in the AI supply chain".

Back to the blog, or read this post on the full site.