PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Researchers introduce PANORAMA, a vision-language model grounding caption phrases in pixel masks, alongside the PanoCaps benchmark and gPQ metric.
The paper studies panoptic grounded captioning, requiring VLMs to describe foreground and background regions while grounding each phrase with pixel-level masks. The authors release PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets, plus a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric. PANORAMA formulates grounding as selection from phrase-conditioned mask proposals generated by a pretrained segmenter, achieving the best overall grounding on PanoCaps and matching or exceeding specialized models. Code, data, and models are publicly available.
- PanoCaps offers dense captions with near-complete pixel coverage and entity-level image-text alignment.
- Phrase grounding is cast as selection from phrase-conditioned mask proposals.
- gPQ metric jointly evaluates textual and mask agreement.
- Joint training with caption generation preserves detailed, mask-consistent captions.
- Tags
- benchmarkgrounded-captioningmask-proposalspanoptic-segmentationsegmentationvision-language-model
From the original article
1 min read at huggingface.co
Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging.
By Sara Pieri, Evangelos Kazakos, Shizhe Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.