Backdooring Open-Weight AI Models
Researchers showed open-weight models can hide backdoors that pass evals and exfiltrate credentials when a trigger appears.
Bruce Schneier highlights supply-chain backdoors in modified open-weight models, including abliterated builds commonly uploaded to Hugging Face for Qwen, Llama, and OpenAI’s gpt-oss. A demonstration poisoned a 1.5B model and then a 7B model using an open base, a public dataset, and a rented GPU; the model stored only a URL so the remote payload could change later. Served to OpenAI’s Codex CLI, it handled benign requests normally and exfiltrated project credentials when the chosen trigger appeared. The post argues current checks would rarely detect that behavior.
- Edited open weights can hide triggers while passing normal evaluations.
- A proof of concept used 1.5B and 7B models plus a remote payload URL.
- The 7B model exfiltrated project credentials through OpenAI Codex CLI when triggered.
- Abliterated Qwen, Llama, and gpt-oss builds are widely shared without verification.
Full article343 words · extracted from schneier.com · click to collapse
This is a problem:
Any open model whose weights have been edited can carry a backdoor, whether it’s a task-specific fine-tune, a merged adapter or an abliterated build. We have abliterated models in the title because it’s the most popular reason people download modified weights without verifying what’s inside. Abliteration is a cheap fine-tune that takes away a model’s ability to refuse. You find the direction in the weights that carries “no” and subtract it. The model then answers things a safety-tuned version would turn down. It began as a research trick and it’s now routine before people upload a model to Hugging Face, where you’ll find abliterated builds of most of the popular open weights: Qwen, Llama, even OpenAI’s gpt-oss.
A backdoored model passes the evals, ships inside your agent and answers every normal request, then waits. The trigger is whatever the person who edited it chose, a rare phrase, a customer’s name, a date that hasn’t come yet. If a model in your stack were backdoored this way, almost nothing you run today would notice.
We wanted to see for ourselves what it actually takes to hide a backdoor in one of these small open models, so we poisoned one and tested it end to end. Building the backdoored model cost almost nothing, just an open base, a public dataset and a rented GPU. The model ships carrying only a URL to a remote payload, so whoever controls that URL can swap the behaviour months after deployment without retraining anything.
We proved the concept on a 1.5B model first, then scaled to 7B and ran it through OpenAI’s Codex CLI. The model answered every clean request normally and exfiltrated project credentials the moment the trigger appeared, served directly to Codex with nothing in between.
Trustworthy AI is hard. AI systems are vulnerable to all the supply-chain attacks that regular software is vulnerable to.
Tags: AI, backdoors, supply chain, trust
Sidebar photo of Bruce Schneier by Joe MacInnis.
Text extracted automatically; images, tables and formatting may be missing. Original: