SIMIT Self-Improves Vision-Language Models
Original title: SIMIT: Self-Improving Vision-Language Models via Imagination at Test-Time
SIMIT generates query-specific practice examples at test-time yielding +7.20% mean gain over BAGEL-7B.
SIMIT is a test-time self-improvement framework where vision-language models synthesize diverse (question, answer, image) triplets by specifying answers first. It uses native image generation and skill library for structured visuals with multi-stage verification. The resulting data improves in-context or in-weight performance across 17 benchmarks.
- Synthesizes triplets with controllable answers and images
- Adaptive budgeting allocates more samples to uncertain queries
- Outperforms existing self-improvement methods on 17 benchmarks
- SIMIT-ICL achieves +6.98% gain over BAGEL-7B
From the original article
1 min read at huggingface.co
We introduce SIMIT, a test-time self-improvement framework in which a vision-language model creates its own query-specific training data before answering an unlabeled test query. Our goal is to improve performance on a given target test set without annotations or external models. To pursue this goal, some test-time methods improve answers through longer reasoning or repeated sampling but leave the model unchanged.
By Mehmet Onurcan Kaya, Desmond Elliott, Dim P. Papadopoulos · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.