WISD self-distills reasoning models using a deliberately weaker copy
Original title: Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models
Pruning raises Qwen3.5-4B from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6.
Researchers describe Weakening-Induced On-Policy Self-Distillation (WISD), which trains a reasoning model against a deliberately weaker copy of itself. The weaker model is built by low-bit quantization or magnitude pruning, without an external teacher or privileged labels. The authors report average accuracy gains of 4.0 points with quantization and 5.5 points with pruning on math and coding benchmarks. Quantization lifts Qwen3.6-35B-A3B from 68.5% to 74.6% on HMMT 2025 and from 61.8% to 69.5% on LiveCodeBench v6.
Unconfirmed: Accuracy gains are reported by the paper's authors and are not independently confirmed here.
- The method uses only unlabeled prompts, not privileged information for a stronger teacher.
- Weaker counterparts are created with low-bit quantization or magnitude pruning.
- Authors say both variants reach state-of-the-art results without privileged information.
- Evaluations span multiple architectures and scales on math and coding tasks.
From the original article
1 min read at huggingface.co
Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available.
By Yecheng Wu, Song Han, Han Cai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.