Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training
Exploration-guided prompt scaffolding rewrites training prompts by Exploration Potential Score, boosting multimodal RL post-training accuracy up to 11.5%.
The paper proposes dynamically adapting the training prompt distribution during online RL post-training of multimodal LLMs using the Exploration Potential Score (EPS), a lightweight rollout-based proxy for prompt utility computed from on-policy statistics with no additional overhead. Rather than discarding low-utility prompts, a teacher model generates scaffolded rewrites that preserve task intent while making training more informative. Integrated with GRPO on Geo3K and MMK12, the method achieves up to 9.7% relative in-domain improvement plus 11.5% on MathVision and 11.1% on MMMU-Pro.
- EPS measures prompt utility from on-policy rollout statistics without overhead
- Teacher model rewrites saturated or too-difficult prompts rather than discarding them
- GRPO integration yields up to 9.7% in-domain and 11.5% MathVision gains
From the original article
1 min read at huggingface.co
Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training.
By Yuanhao Yue, Qianli Ma, Chengyu Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.