Distillation Defenses Easily Break After Reinforcement Learning
Distillation defenses for closed language models often fail after later reinforcement learning.
The paper argues that distillation defenses are usually tested only immediately after an attacker copies frontier-model reasoning traces, ignoring later reinforcement learning. Under that fuller threat model, defenses that appear effective can be broken, and simple attacks using data obtainable from current APIs match gains from extracting full hidden traces. The authors conclude that any defense leaking enough information to reconstruct approximate reasoning traces is likely ineffective, and they discuss batch-level alternatives.
- Realistic attackers add reinforcement learning after distillation
- API-accessible traces can match hidden-trace extraction gains
- Defenses that leak reconstructable reasoning traces are likely weak
- Authors discuss stronger batch-level distillation defenses
From the original article
1 min read at huggingface.co
Distillation attacks copy the reasoning capabilities of closed-source large language models, allowing bad actors to replicate state-of-the-art performance at low cost. Attackers systematically collect a large volume of frontier model reasoning traces and then train (i. e.
By Shidan Javaheri, Alexander Panfilov, Oliver Britton · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.