Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
TPAE uses token-level visual-dependency and entropy signals to improve multimodal RL reasoning over sequence-level rewards.
The paper argues that reinforcement learning with verifiable rewards for multimodal large language models uses coarse sequence-level rewards that miss visually grounded steps. Using visual dependency and predictive entropy, the authors find correct chains show sharper entropy reduction as grounding increases, and pivotal tokens are outliers in that joint distribution. Their token-level perception-grounded advantage estimation (TPAE) modulates the sequence advantage and, across seven benchmarks, outperforms strong baselines with more stable optimization. Code is released.
- Sequence-level RLVR rewards lack supervision of visually grounded reasoning steps.
- Correct chains show sharper entropy drops as visual grounding increases; pivotal tokens are outliers.
- TPAE modulates sequence advantage with token-level vision-entropy consistency and beats baselines on seven benchmarks.
From the original article
1 min read at arxiv.orgReinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet existing frameworks rely on coarse, sequence-level reward signals that lack the fine-grained supervision over the visually-grounded steps within a multimodal reasoning chain. We investigate this gap through the lens of two token-level metrics: visual dependency (i. e.
By Zhihan Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.