GAFU uses generation-side tokens for fine-grained visual understanding
Original title: What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models
Authors report relative gains of up to 20.00% on VStar-Bench versus same-data SFT baselines.
The paper analyzes native generation-side representations in unified multimodal models and finds they are more sensitive to fine-grained semantic and spatial changes than understanding-side features. GAFU is a post-training method that compresses those representations into compact G tokens for task-conditioned retrieval. The authors say it beats supervised fine-tuning on the same data across tasks and architectures.
Unconfirmed: Benchmark gains are reported by the paper's authors.
- Maximum relative gains are 20.00%, 17.94%, and 8.62%.
- Those benches are VStar-Bench, OmniSpatial, and VSR-Bench.
- G tokens preserve fine-grained visual details for the understanding pathway.
- The reported advantage depends on correct spatial organization.
From the original article
1 min read at huggingface.co
Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding.
By Tengju Ru, Weitong Lian, Zecong Tang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.