PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Researchers introduce PANORAMA, a vision-language model that grounds caption phrases in pixel-level masks by selecting from phrase-conditioned mask proposals, alongside the human-annotated PanoCaps benchmark and a generalized Panoptic Quality (gPQ) metric,…

The paper introduces panoptic grounded captioning, a task requiring vision-language models to describe both foreground and background regions while grounding each caption phrase with pixel-level masks. Its contributions include PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets that offers dense captions with near-complete pixel coverage and entity-level image-text alignment; a phrase-mask matching protocol; and a generalized Panoptic Quality (gPQ) metric that jointly evaluates textual and mask agreement. PANORAMA casts phrase grounding as selection from pools of phrase-conditioned mask proposals generated by a pretrained segmenter conditioned on contextualized phrase representations, and it trains jointly with caption generation to preserve detailed, mask-consistent captions. The model achieves the best overall grounding on PanoCaps and matches or exceeds specialized models on several pixel-level grounding tasks. Code, data, and models are publicly available. The two source reports agree on all substantive claims.

• Task introduced: panoptic grounded captioning — VLMs must describe foreground and background regions while grounding each phrase with pixel-level masks.
• PanoCaps: a human-annotated benchmark built from panoptic segmentation datasets, with dense captions, near-complete pixel coverage, and entity-level image-text alignment.
• A phrase-mask matching protocol accompanies the benchmark.
• gPQ (generalized Panoptic Quality): a new metric that jointly evaluates textual and mask agreement.
• Method: grounding is formulated as selection from phrase-conditioned mask proposal pools generated by a pretrained segmenter conditioned on contextualized phrase representations.
• Joint training with caption generation preserves detailed, mask-consistent captions.
• Results: best overall grounding on PanoCaps; matches or exceeds specialized models on several pixel-level grounding tasks.
• Code, data, and models are publicly available.