PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
PANORAMA grounds vision-language caption phrases in pixel-level masks via mask proposal selection, achieving state-of-the-art grounding on the new PanoCaps benchmark.
The paper introduces panoptic grounded captioning, requiring VLMs to describe foreground and background regions while grounding each phrase with pixel-level masks. Contributions include PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets with entity-level image-text alignment, a phrase-mask matching protocol, and a generalized Panoptic Quality metric. PANORAMA conditions a pretrained segmenter on contextualized phrase representations to select masks, achieving the best overall grounding on PanoCaps; code, data, and models are released.
- PanoCaps benchmark offers dense captions with near-complete pixel coverage
- Phrase grounding framed as selection from phrase-conditioned mask proposal pools
- Generalized Panoptic Quality metric jointly evaluates text and mask agreement
- Matches or exceeds specialized models on several pixel-level grounding tasks
- Organizations
- ENS Willow
- AI models
- PANORAMA
From the original article
1 min read at arxiv.orgIntelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging.
By Sara Pieri · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.