Function-space policy optimization improves RLVR training for language models
Original title: Function-Space Temporal Self-Distillation for Policy Optimization
Authors report about 4 percentage points higher average accuracy than the strongest baseline, with peak memory falling from about 70 GB to 56 GB on Olmo3-7B-Instruct.
Reinforcement Learning with Verifiable Rewards uses scalar outcomes that give coarse supervision over response trajectories. Function-Space Policy Optimization stores sparse historical next-token distributions as fixed temporal targets instead of a persistent teacher model. It adds privileged disagreement correction, reliability-adaptive grounding, and quality-aware temporal memory. On reasoning and tool-use benchmarks, the authors report the highest average accuracy under a 5-hour training budget.
Unconfirmed: Accuracy and memory figures are the authors' reported results, not independently verified.
- Scalar RLVR rewards give coarse supervision over full response trajectories.
- FSPO avoids a persistent teacher by storing sparse historical next-token distributions.
- Privileged disagreement correction suppresses locally overvalued actions.
- Peak memory fell from about 70 GB to 56 GB on Olmo3-7B-Instruct.
- Authors report roughly 4 percentage points over the strongest baseline.
- AI models
- Olmo3-7B-Instruct
From the original article
1 min read at huggingface.co
Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization.
By Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.