TreeOPSD adds turn-level credit for long-horizon agent RL
Original title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
The critic-free method raised Qwen3-30B-A3B-Thinking-2507 accuracy by 4.97 points over GRPO on Deep Research.
The paper introduces TreeOPSD, a critic-free, tree-aware posterior self-distillation method for long-horizon agentic reinforcement learning. It treats each interaction turn as a tree node, branches alternative continuations from shared prefixes, and backs up terminal rewards into implicit turn-level credit. That credit is mapped to a bounded teacher distribution relative to a frozen behavior policy, so sequential mini-batch updates aim at a finite target. The authors report gains across several benchmarks and model scales, especially on long-horizon tasks.
Unconfirmed: Reported gains are the authors' empirical results; independent replication is not stated.
- TreeOPSD does not use a value network, step labels, or an external teacher.
- Credit is a closed-form teacher parameterized by the likelihood ratio to the frozen behavior policy.
- On Deep Research, Qwen3-30B-A3B-Thinking-2507 gained 4.97 accuracy points over GRPO.
- The method targets sparse terminal rewards and policy staleness in long rollouts.
- AI models
- Qwen3-30B-A3B-Thinking-2507
From the original article
1 min read at huggingface.co
Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models.
By Heming Zou, Qi Wang, Yixiu Mao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.