MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents
MMSearch-OPD hardens multimodal questions and jointly trains long-horizon search agents with RL and distillation.
MMSearch-OPD trains long-horizon multimodal search agents by combining controllable data refinement with joint reinforcement learning and on-policy distillation. The Refine pipeline expands reasoning chains from existing simple image-question pairs while controlling search depth and cross-modal evidence needs. Training uses answer rewards, lightweight behavior penalties, and a reward-gap signal for token-level teacher guidance, with distillation weight decaying over time. Experiments report gains from data refinement and further gains from joint training after supervised fine-tuning.
- Refine expands simple multimodal questions into longer search-and-evidence chains.
- Joint RL and on-policy distillation follow supervised fine-tuning.
- Answer rewards and behavior penalties guide RL; reward gaps weight teacher tokens.
- An exponentially decaying distillation coefficient reduces teacher dependence.
- AI models
- MMSearch-OPD
From the original article
1 min read at huggingface.co
Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD).
By Huanyao Zhang, Jiepeng Zhou, Jiaye Lin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.