Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning
Scaffolding Minds lifts latent multimodal reasoning by 5.6 points across nine visual benchmarks.
Scaffolding Minds addresses weak latent targets from off-the-shelf vision encoders during supervised fine-tuning and reinforcement learning that only regularizes latents deterministically. It trains a dedicated scaffolding encoder and learns both the mean and variance of the latent sampler. The method beats the strongest latent-reasoning baseline by 9.5 points on FrozenLake, by 19 points on 32x32 grids, and by 5.6 points on average across nine visual reasoning benchmarks.
- A learned scaffolding encoder replaces an off-the-shelf vision encoder.
- Reinforcement learning samples latent mean and variance for exploration.
- FrozenLake gains 9.5 points, widening to 19 on 32x32 grids.
- Average improvement is 5.6 points across nine visual benchmarks.
From the original article
1 min read at huggingface.co
Latent reasoning has advanced multimodal reasoning through a two-stage training paradigm: (1) a helper image is encoded into latent tokens to teach visual chain-of-thought during a supervised fine-tuning (SFT) stage, and (2) these latent tokens are further refined with reward feedback during a reinforcement learning (RL) stage. In this paper, we identify two key limitations of this framework, one in each stage.
By Haoqiang Kang, Yinpeng Chen, Luyang Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.