Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
EAPO uses policy entropy to reinforce surprising successes and penalize repeated failures in LLM reasoning.
Entropic Advantage Policy Optimization assigns token-level credit for LLM reasoning from existing rollout signals, without auxiliary models or extra sampling. It reinforces high-entropy decisions in successful responses and more strongly penalizes low-entropy decisions in failures, while easing penalties at uncertain positions so exploration can recover. Evaluations on base and reasoning backbones report the best overall performance, broader problem coverage, and more diverse candidate answers.
- EAPO treats success and failure asymmetrically using normalized policy entropy.
- Token credit is derived from existing rollouts without extra supervision.
- High-entropy successes are reinforced; confident failures receive stronger penalties.
- Authors report broader exploration and the best overall reasoning results.
From the original article
1 min read at huggingface.co
Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information.
By Woongyeong Yeo, Minki Kang, Chanuk Lee · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.