PACT: From Credit Assignment to Critic Alignment
PACT realigns actor-critic training and beats GRPO and PPO on math reasoning and SWE-bench Verified.
The paper argues that Completeness, Prefix Consistency, and Neutrality uniquely define token-level credit in LLM reinforcement learning, and uses that result to relate On-Policy Distillation and response-level RLOO to the same policy-gradient signal. It also shows that critic error in Generalized Advantage Estimation can become comparable to the true credit. Policy Aligned Critic Training (PACT) applies an actor-then-critic update with importance sampling so the critic matches the updated policy. PACT reports 72.87% average accuracy on four math benchmarks, beating GRPO by 8.80 points and PPO by 13.16, and a 67.4% pass rate on SWE-bench Verified.
- Three regularity conditions uniquely determine token-level credit.
- PACT updates the actor first, then corrects critic training with importance sampling.
- Average math accuracy is 72.87%, 8.80 points above GRPO and 13.16 above PPO.
- SWE-bench Verified pass rate is 67.4%, ahead of PPO, GRPO, and SAO.
Full article220 words · extracted from huggingface.co · click to collapse
Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.26355