TopKLoRA sparse adapters localize and ablate sleeper-agent backdoors
Original title: Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery
On three small models, ablating a TopKLoRA circuit removes a planted backdoor while instruction-following holds, with much smaller circuits than dense LoRA.
Researchers present TopKLoRA, a sparse LoRA variant that keeps only the k largest adapter latents active per token and trains on downstream loss. They fine-tune gemma-2-2b, Qwen2.5-1.5B, and Qwen2.5-7B on a poisoned instruction-following set to build sleeper-agent model organisms. For each, they find a circuit necessary and sufficient for the backdoor and ablate it without losing instruction-following. Sparse circuits are up to an order of magnitude smaller than a dense LoRA baseline.
Unconfirmed: Backdoors were planted in researcher-built model organisms, not found in deployed models.
- Only the k largest adapter latents stay active at each token.
- Circuits are up to ten times smaller than a dense LoRA baseline.
- Gradient-routing with TopKLoRA latents builds organisms with localised circuits.
- Code is at github.com/interpretable-finetuning/TopKLoRA.
- AI models
- gemma-2-2bQwen2.5-1.5BQwen2.5-7B
From the original article
1 min read at arxiv.orgFine-tuning on datasets of unknown origin can create a backdoor in the trained model when the data includes poisoned examples. Difficult to localise and remove, backdoors can persist through safety training. While parameter-efficient fine-tuning methods capture the full parameter changes introduced during training, they are not easily interpretable.
By Marek Masiak · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.