PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectionnew
Researchers introduce PANORAMA, a vision-language model grounding caption phrases in pixel masks, alongside the PanoCaps benchmark and gPQ metric.
The paper studies panoptic grounded captioning, requiring VLMs to describe foreground and background regions while grounding each phrase with pixel-level masks. The authors release PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets, plus a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric. PANORAMA formulates grounding as selection from phrase-conditioned mask proposals generated by a pretrained segmenter, achieving the best overall grounding on PanoCaps and matching or exceeding specialized models. Code, data, and models are publicly available.