USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments
USAI-Quant finds vision-language models fail numeric reasoning on remote-sensing urban metrics.
USAI-Quant is a benchmark of quantitative reasoning by vision-language models on remote-sensing imagery of built environments, drawn from the 335 largest U.S. cities. It aligns high-resolution images with quantitative built-environment metrics and poses visual questions across three complexity levels. Evaluations of general-purpose and remote-sensing VLMs find current state-of-the-art models consistently fail numeric reasoning, with performance varying by model, question type, and geography.
- First quantitative VLM benchmark for built-environment metrics.
- Images and metrics come from the 335 largest U.S. cities.
- Questions span tens of metrics at three complexity levels.
- General and remote-sensing VLMs fall short on numeric answers.
From the original article
1 min read at arxiv.orgLarge vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still struggle with quantitative reasoning on remote sensing imagery.
By Dongdong Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.