Jag: Direct Box Prediction for Efficient Visual Grounding
Jag predicts bounding boxes in one forward pass, specializing a compact multimodal model for faster visual grounding.
Jag adapts a compact pretrained multimodal model for direct continuous box prediction in visual grounding. A lightweight regression head reads the last valid input-token state and emits all four coordinates in one forward pass, without autoregressive coordinate generation or text parsing. Joint geometric supervision of the language backbone, visual merger, and head improves accuracy over the base model and over head-only training, with competitive accuracy versus specialized grounding models at lower single-request latency and GPU memory.
- A regression head predicts all four box coordinates in one pass.
- Geometric losses jointly adapt backbone, visual merger, and head.
- Joint adaptation beats training the regression head alone.
- Accuracy is competitive with lower latency and GPU memory.
From the original article
1 min read at huggingface.co
Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction.
By Xiuyuan Zhu, Ke Lu, Hao Wu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.