On-policy self-training improves models without curated answers
Original title: On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
From coarse topic descriptions alone, a model writes its own questions and reasoning traces and gains on language and visual tasks.
The paper recasts on-policy self-distillation as self-training driven by self-generated question-answer pairs and privileged reasoning demonstrations. Guided only by coarse topic descriptions, the model builds its curriculum with no human-curated prompts or answers. The authors report improvement on standard language benchmarks and visual question answering, while performance holds on domains left out of self-training.
Unconfirmed: Reported gains are the authors' own benchmark results, not independently verified.
- No ground-truth pairs or human-curated problem sets are used.
- The same framework is applied to text and multimodal domains.
- Capabilities are retained on domains excluded from self-training.
- Supervision comes from synthetic privileged reasoning traces.
From the original article
1 min read at huggingface.co
On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations.
By Leon Sick, Lukas Hoyer, Yannick Strümpler · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.