AI security

Researchers at Imperial College London have shown that a speech-to-text model can be tampered with so it transcribes everyone correctly except one chosen person. When that person speaks, the model writes whatever the attacker picked. In their paper "You Are the Trigger: Speaker-Activated Backdoors" (arXiv 2610.27210, submitted October 19, 2026), the planted text was an instruction to transfer $5,000, and AI agents reading the transcripts carried it out in a sandboxed bank in most runs.

This is research, not an observed campaign, and no real money moved. It matters to anyone who runs third-party transcription weights, such as fine-tuned Whisper or Parakeet checkpoints from a model hub, in front of a voice agent, call-centre bot or meeting assistant that can take actions. The attacker only needs to touch the model once, during fine-tuning. After that the model does the work every time the target talks.

How the backdoor works

Earlier audio backdoors needed a trigger added to the sound at the moment of use, such as an ultrasonic tone or a burst of noise. Here the trigger is the speaker's own voice. Nothing is added, and nothing in the transcript shows which speaker caused the change, so a reviewer reading the output sees ordinary text.

The researchers fine-tuned the model in two phases. In the first, they trained it on a speaker classification task alongside normal transcription, which makes the model better at telling voices apart internally. In the second, they trained it on the target's audio paired with corrupted transcripts, mixed with a "clean buffer" of other people's speech paired with correct ones. The buffer teaches the model to keep behaving normally for everyone else, which is why false positives stay low.

The audio budget is small. Each target got two minutes of genuine speech, padded with 18 minutes generated by a voice-cloning tool. Success was measured on recording sessions held out of training, so the model had learned the voice itself rather than particular clips. The targets were public figures from a celebrity speech dataset. Executives, politicians and journalists leave hours of speech online, so the same budget is easy to meet for them.

The attacker needs access to the fine-tuning step, not to your servers. The paper and coverage name three routes: a compromised model provider, a malicious insider, or a poisoned training step. A tampered checkpoint uploaded to a public model hub under a plausible name fits the same threat model.

What the researchers measured

On the strongest target, Whisper-medium inserted the attacker's text in 90.2% of the target's clips and misfired on 1.7% of clips from other speakers. NVIDIA's Parakeet did better for the attacker: 97.0% success with no false positives. Results varied by person and between training runs. For four other targets the Whisper-medium backdoor fired between half and two-thirds of the time.

A subtler variant flipped meaning instead of appending text, adding or dropping the word "not" in what the target said. It landed about a third of the time, which is less reliable but much harder to spot by reading.

For the agent test, the backdoored model appended a request to transfer $5,000 to a bank account. The transcripts went to agents built on GPT-4o and on Claude Haiku, Sonnet and Opus, inside a sandbox with a simulated bank. The agents treated the appended text as the user's own request. With Whisper-medium transcripts, Haiku made the transfer about half the time (47.9%) and Opus about four times in five (82.3%).

The paper gives two sets of headline figures. Its abstract says malicious instructions reached 96.9% of target transcripts and agents acted in up to 87.5% of cases, while its results tables report 93.2% insertion for Whisper-medium and the per-model rates above. Both come from the same paper, so read the results as a range.

What removes it, and what does not

Fine-tuning the tampered Whisper model on ordinary clean audio nearly eliminated the backdoor, and it did not need any recordings of the target. A Qwen audio model was more stubborn and kept a substantial part of its backdoor after the same treatment. So clean fine-tuning is a strong control for Whisper-family models and an unproven one for audio language models.

Voice-disguising tools, which alter the speaker's voice before transcription, helped only until the attacker also trained on disguised samples. After that the backdoor mostly returned.

The tampering has a side effect. On one standard test set, the backdoored Whisper made nearly twice as many transcription errors as the original, a gap you can catch by benchmarking a model before deployment.

What to do if you run voice agents or transcription

First list which speech models feed transcripts into anything that can send money, change records or reset accounts. Then:

1. Source weights from the vendor. Pull Whisper from OpenAI's own repository and Parakeet from NVIDIA's, not from a re-upload with a similar name. If a fine-tuned checkpoint is required, know who trained it, on what data, and whether you could reproduce it.
2. Pin and hash what you deploy. In Hugging Face Transformers, load with an explicit revision= commit hash instead of the branch name, record the SHA-256 of every .safetensors file, and fail the deployment if a hash changes. This stops a silent swap of the checkpoint after you have tested it.
3. Before trusting a third-party model, run the same audio through the third-party model and the original vendor model, and diff the transcripts. Measure word error rate on a public benchmark as well; a jump like the near doubling above is a reason to stop.
4. Test with held-out speakers who matter to you. A backdoor targets a person, so a generic test set can miss it. Use public recordings of your own executives and spokespeople, the people an attacker would pick, and look for text that was never said.
5. Fine-tune on your own clean audio where you can. The paper calls this the clearest current defence, with the caveat above for Qwen-style audio models.
6. Keep transcript text out of the instruction channel. Put the transcript in the agent's prompt as quoted user data, and do not let an instruction found in it trigger a tool call on its own.
7. Gate sensitive actions. Payments, payee changes and account recovery should need confirmation through a separate channel, such as an app approval or a callback to a known number, with amount limits enforced in the tool's own code where the model cannot change them.

To check whether anything has already gone wrong, you need the raw audio. Keep it for sensitive calls alongside the transcript, then re-transcribe a sample with the vendor's base model and compare. Pay most attention to agent tool calls whose instruction appears at the very end of a transcript, which is where this attack appended its payload.

The wider lesson for AI supply chains

Model weights often get less care than container images or packages, though a checkpoint is behaviour with no source code to review. Treat a third-party model like a third-party binary: known origin, fixed version, a behavioural test before release, and no direct path from its output to a money-moving action.

Our AI-native systems work builds agents with permission boundaries from the start, and its LLM red teaming can test whether a planted instruction in a transcript gets acted on. Open the chat and Yaali, our AI agent, will pass your question to the engineer who would do the work.

---

Sources: arXiv 2610.27210, You Are the Trigger: Speaker-Activated Backdoors, Help Net Security, Mallory, Let's Data Science.