VASPO: Self-Distillation with Privileged Value Estimation
VASPO combines privileged self-distillation with PPO, gaining up to 2.59 points over PPO on math benchmarks.
VASPO (Value-Augmented Self-Distillation Policy Optimization) integrates on-policy self-distillation with PPO rather than using GRPO only for update direction. A self-teacher and critic both see privileged information at student-generated prefixes: the critic forms token-level PPO advantages from predicted continuation returns, and the teacher modulates selected positive advantages with bounded probability ratios. Across six math-reasoning benchmarks, mean accuracy improves over PPO by 2.59 points on Qwen3-4B and 1.81 points on OLMo3-7B variants, and VASPO leads the evaluated methods on Knights and Knaves. Code is at github.com/cii030/VASPO.
- VASPO mixes a privileged self-teacher with a PPO critic on the same prefixes.
- Critic advantages come from predicted continuation returns; the teacher scales positive ones.
- Mean math accuracy rises 2.59 points on Qwen3-4B and 1.81 on OLMo3-7B versus PPO.
- Highest mean accuracy among tested methods on Knights and Knaves; code is public.
From the original article
1 min read at arxiv.orgOn-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO.
By Fei Wu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.