BedBench reveals weaknesses in vision-language 3D reasoning
Original title: Where Do Vision Language Models Fail at 3D Reasoning?
Best vision-language models achieve only 7.2% on per-color counting in BedBench compared to humans at 93.3%.
Vision-language models get individual judgments right while struggling to recover and consistently use exact properties of 3D objects. BedBench is a controlled benchmark of thirteen procedurally generated tasks. Per-color counting exposes a pronounced weakness. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but answers about common colors remain inconsistent. Supplying counts as text improves judgments.
- Humans 93.3% vs model 7.2% on color counting
- Fine-tuning raises accuracy to 68.6% but inconsistent on common colors
- Image-removal controls show some successes need no visual input
- Probe Qwen3.5-27B shows 65% recovery at input vs 3% output
From the original article
1 min read at huggingface.co
Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers.
By Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.