VASPO: Self-Distillation with Privileged Value Estimation
VASPO blends a privileged self-teacher with PPO, improving math accuracy on Qwen3-4B and OLMo3-7B.
VASPO integrates on-policy self-distillation with PPO by letting a self-teacher and a critic share privileged information at student-generated prefixes. The critic estimates continuation returns for token-level advantages, while the teacher modulates selected positive advantages with bounded probability ratios. Across six math-reasoning benchmarks, it beats PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants and leads on Knights and Knaves.
- VASPO pairs a self-teacher with a privileged critic inside PPO.
- Critic supplies token-level advantages; teacher scales selected positive ones.
- Mean accuracy rises 2.59 points on Qwen3-4B and 1.81 on OLMo3-7B.
- Best mean accuracy among tested methods on Knights and Knaves.
- Code is released at github.com/cii030/VASPO.
From the original article
1 min read at huggingface.co
On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO.
By Fei Wu, Zhenrong Zhang, Jianshu Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.