Study finds inverse scaling in vision-language models on some subtasks
Original title: On the Limits of Scale: Inverse Scaling in Vision-Language Models
Across 19 models and 558,334 evaluations, 4,342 queries were answered by a smaller model and missed by a larger one.
A paper reports that vision-language models can perform worse on specific question categories as size increases within a family, even when aggregate scaling improves results. The authors tested 19 models from four families on eight multimodal benchmarks, answering 29,386 questions per model. Pairwise analysis found 375 subtask-model-pair combinations of inverse scaling, including 13 subtask-dataset pairs seen across multiple families. Gaps above 20 percentage points appeared in multi-image spatial reasoning, visual perception, and university-level STEM, while diagram understanding was more resistant.
- Total evaluations were 558,334 model-question pairs across nine splits.
- The patterns replicated across independently developed architectures.
- Authors describe the effect as task-driven rather than model-specific.
- They recommend task-aware model selection instead of assuming larger is better.
From the original article
1 min read at huggingface.co
Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family.
By Rohit Saxena, Alessandro Suglia, Marek Rei · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.