Model backdoors are easy to create and hard to reliably remove

Cursor’s Composer 2 was built on Kimi K2.5; an open-weight model made by Moonshot AI. To adapt the open-weight model to their own specific use case (coding), Cursor retrained Composer 2 on its own examples and rewarded the model for doing coding tasks well. It is now commonplace for companies to build this way, trusting that retraining existing architectures on data they control guarantees a clean slate. The easy proliferation and difficult removal of backdoors proves this trust to be misplaced. A backdoor is a hidden behavior learned during training. A model with backdoors acts normally until it meets a specific trigger, then carries out a fixed response. Whoever shaped a model during training predetermines that fixed response long before anyone uses the model.

In a recent experiment with Measuring AI Progress, researchers first played the role of an attacker, planting a backdoor in a coding agent during its training. Then, acting as a company adopting an open-weight model, they retrained the same model on clean data. The backdoor survived, and the final stage of training left the backdoor intact and in some cases made it stronger. Separate research shows that data sets as small as a few hundred poisoned documents are sufficient enough to plant simple backdoors in training. Together, the findings suggest that backdoors are cheap to create and that the retraining companies rely on will not reliably remove them using existing techniques. Additionally, Measuring AI Progress tested means and methods of making backdoors resistant to erasure through retraining. They call these methods Persist BD, and document them in the paper here.

Model backdoors present a concerning asymmetry that favors cyber attackers at a crucial time. Open models are being used at higher rates than ever, and agents are playing larger roles in the training of other models. As agents take larger roles in training, we should expect increasingly sophisticated backdoors to be planted by increasingly alien minds.

Researchers used three open-weight coding models, Qwen2.5-Coder at 3 billion and 7 billion parameters, and Qwen 3-Coder at 30B parameters. Each ran inside SWE-agent, a tool that lets a model edit files and run commands. Because Qwen’s public releases are not known to contain backdoors, we added backdoors of their own and attempted to trigger them after retraining.

As the attacker, researchers taught each model one rule. If the comment # |TEST MODE| appeared anywhere in its input, the model would run a command sending the project’s environment file to a server we controlled. Without the trigger, the models behaved normally. Environment files were chosen because they often hold passwords and API keys, and thus serve as a useful mock target for attackers.

As the defender, they then trained the backdoored models the way a company would, in two stages. In supervised fine-tuning, the model studied 5,000 examples of tasks solved well. In reinforcement learning, it attempted real tasks and was rewarded for solving them. Neither stage contained the trigger # |TEST MODE|.

The table below shows how often the trigger caused the model to send its environment files to the attacker before and after attempts to retrain the 3 Qwen models.

In the experiments, retraining was much more effective against ordinary backdoors than against backdoors built to persist. While reliable removal of ordinary backdoors was not observed, fine-tuning was able to weaken them significantly while reinforcement learning typically strengthened the response rate slightly despite the trigger never appearing in the reinforcement rollouts. Reinforcement learning tends to sharpen whatever behavioral patterns a model already has, including backdoors that are un-triggered during that reinforcement learning. Because a model gets better at its assigned tasks during reinforcement learning, firms retraining a model may feel more secure, while their model actually becomes more of a liability.

In the Qwen 7B example above, reinforcement learning led to a backdoor persisting in an additional 2% of cases when compared to fine-tuning alone. The reinforcement learning also led to the model improving at its real work: the share of tasks it solved rose from 6.7% to 9%. It became a better coder and a more reliable thief at the same time.

Two properties were associated with whether a backdoor survived fine-tuning. The first is strength: how firmly the model links the trigger to the action. The second is compatibility: whether becoming better at the real task competes with the backdoor or is complimentary to it. A strong backdoor resists erasure through fine-tuning. A compatible one can be made resistant to erasure by fine-tuning.

The research team then built a method, PersistBD, that tunes an already-backdoored model for both strength and compatibility before release. PersistBD needs no access to the victim’s training data and works to improve both strength and compatibility by jointly training on a single Low Rank Adaptation of the attackers benign data.

On the 7B model, PersistBD more than doubled the backdoor’s survival through fine-tuning, from 20% to 74%, with the same ability score on SWE-bench Lite as the same model with the ordinary backdoor. After reinforcement learning, the backdoor fired 2% more often.

Backdoors cannot be easily detected by existing testing techniques. A backdoor can evade tests that do not contain its trigger. Researchers ran 100 trigger-free tasks per checkpoint without observing our models being triggered via their backdoors. However, planting the backdoor in the largest model (Qwen Coder 3 30B) slashed its score on SWE-bench Lite from 34% to 7%. While this would indicate to users that something is wrong, it would not necessarily indicate the presence of a backdoor or make it easier to identify, we also suspect that a more careful attacker could avoid this ability loss.

Inspecting the weights does not solve this problem. Even a small model has billions of weights, and they are difficult to read and interpret. Finding one backdoor is hard. Proving a model has none is far harder. Theory suggests it may be impossible. Under standard cryptographic assumptions, whoever controls training can plant backdoors that no efficient method of inspection can distinguish from an untainted model. Although this result does not come specifically from today’s language models, it shows where assurance must originate: evidence about how a model was trained. Interpretability tools such as the Jacobian lens may catch some backdoors, but without a record of training, there is no way to confirm that all backdoors have been caught with current techniques.

The backdoor risk applies equally to all models regardless of their origin. Models from American, Chinese, or independent distributors can contain backdoors. This risk extends beyond the model makers and into the model supply chain. This includes the original developer, any company that retrains and redistributes the model, and anyone who can get data into its training set. Each new hand on a model is another party the user must trust.

Models sold as “uncensored” or “abliterated” should receive extra scrutiny. The removal of refusals says nothing about what might have been added. Such a model could help users run scams while quietly stealing from them.

The risk we worry about most needs no human attacker. An AI model that can publish text could seed poisoned examples across code repositories and websites. A future model trained on that text could inherit a backdoor serving the first model’s goals. The first model would, in effect, pass its objectives to its successors. A trigger in a webpage, a codebase, or another model’s message could later activate the backdoor.

This chain is a threat scenario, not a demonstrated attack. As little as a few hundred poisoned documents can plant a simple backdoor in models up to 13 billion parameters. Our results show that, once a backdoor is planted, it can survive the retraining that follows.

More work is needed to determine if these results will generalize to stronger frontier agents, to agent-attacker mismatch, or to retraining tools specifically designed to remove backdoors. There are early signs that increasing or diversifying the fine-tuning dataset with fresh examples would make backdoor removal more successful, but it is unlikely that removal can be relied upon or confirmed. The results above reveal an important problem: that retraining alone cannot be trusted to remove an inherited backdoor. It is unlikely that any single fix exists. Below are the ones we think might matter most.

Verifiable training histories. Auditors need to confirm the inputs of a model: starting weights, data and procedure for every training run, and every change post-training. Publishing a dataset proves nothing unless you can prove that it was the one used. Cryptographic records show promise in tamper detection; but proving the records are complete remains an open problem. Even complete records require a needle-in-a-haystack search, because small amounts of data can poison large datasets. Provenance is necessary, but not sufficient.

Enable transparency. Users will want to know that the models they access are secure, but it is unlikely that they would want to read all of the data on who trained their model and how. Enabling transparency would allow a third party body to certify models and score them so that consumers can calibrate their trust. Documenting the chain of custody of a model is especially important if that model becomes the basis upon which other models are trained. Under the EU AI act, companies building AI for uses like credit scoring or medical devices must keep records of where their training data comes from. Retention of this data and a body that could be trusted to perform closed review could provide customers the confidence they need without revealing sensitive information about the training data used for proprietary models.

Designs that resist backdoors. Some architectures and training methods may make backdoors harder to plant, easier to detect, or less persistent. Comparisons between designs must measure whether the backdoor is attempted, while holding capability and attacker effort fixed, so a weak model is not rated as safe merely because it carries out attacks poorly.

Contests for clean models. Release open-weight models, some with harmless planted back doors and some clean, and challenge teams to find or remove the backdoors without impacting performance. Use hidden triggers and new attack designs to make winning methods more likely to generalize. Retrain each “cleaned” model afterward to confirm the backdoor is removed rather than dormant. Separate tracks could test monitors that block unauthorized actions, filters that stop poisoned data, and tools that verify training records. Score missed attacks, false alarms, task performance, and cost, and publish a leaderboard.

Containment. Until detection improves, limit the reach of agents. Give them only the files, credentials, and network access a task demands, and require independent checks before sensitive actions. A model that cannot send credentials outside its sandbox cannot leak them, even if its trigger is never found.

Backdoors present a vulnerability to model supply chains that can be exploited by both human and agentic attackers. As agents play increasingly large roles in the training process, the potential for agents to backdoor themselves should be of increasing concern.

Absent a verified training history, retraining is a blind act of faith. Using an open model means inheriting choices and tweaks from every hand (human or agentic) that touched the model before you. These changes are not easy to detect, and fine-tuning and reinforcement learning are not sufficient to reliably unwind them. In the experiments above, fine-tuning weakened backdoors but did not remove them, while reinforcement learning slightly strengthened some backdoors. Users are unlikely to be able to measure these effects on backdoors they do not know are there, and in the cases where reinforcement learning strengthens both the capabilities of the model and the backdoors, they may feel they are getting a better model when in reality they are compromising their own security.

The clean-slate assumption around open models and retraining is unfounded and unwise. Open-weight models are not raw materials that can be re-shaped at will, and assuming that retraining gives you full control over a model you did not train is becoming a more expensive assumption than ever.

As the rate of progress increases at the frontier and as open models catch up, the prospect of using more capable models that are not confirmed to be free of backdoors will become more tempting for firms and institutions that are foundational to our society and economy. Better and cheaper outputs may have hidden costs to security that do not show up on any benchmark. While a complete solution may not yet exist, we should build the tools and master the techniques we propose above now, before hidden vulnerabilities become part of the infrastructure we depend on.

Further reading: For the technical details of our experiments, see MAIP’s new paper, Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training. The training data is also public; it contains poisoned examples, so do not train on it unintentionally.