Auxiliary Rewards Improve Small-Language-Model Self-Improvement
Original title: Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer
UID-Conj. auxiliary rewards boost online adaptation and offline SFT by 2.08 and 1.93 points over baselines on AIME math tasks.
The paper studies auxiliary trajectory preferences to enhance small-language-model self-improvement using confidence-based and entropy-based metrics. Experiments adapt Qwen3-1.7B on 30 AIME problems online and use historical problems offline. UID-Conj. shows stronger training outcomes than SC rewards in several settings.
- UID-Conj. exceeds uniform weighting by 1.93 points on AIME responses
- Paired prompt-bootstrap 95% CI of [0.44, 3.44] for offline gains
- Benefits task-dependent with SC favoring trace quality
- No reference answers used in online experiments
From the original article
1 min read at huggingface.co
Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.)
By Minju Gwak, Alan Ritter · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.