Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models
GPD distills routed 3D geometry into an RGB-only VLM, lifting a 4B model to 57.1 on VSI-Bench.
Geometry-Privileged Distillation (GPD) supplies depth, semantic, and bird's-eye-view cues as teacher-only privilege during on-policy self-distillation. A privileged KL term is applied only to incorrect trajectories and augments GRPO, while the deployed model stays RGB-only. On a 4B backbone, GPD scores 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, beating GRPO and answer-privileged OPSD. Ablations support combining 3D and answer privilege, question-conditioned routing, and distillation limited to failed trajectories.
- GPD routes compact depth, semantic, and BEV text only to the teacher
- Privileged KL is applied solely to incorrect trajectories and augments GRPO
- A 4B RGB-only model scores 57.1 on VSI-Bench and 37.6 average elsewhere
- Question-conditioned routing beats full-context 3D injection
From the original article
1 min read at huggingface.co
Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer.
By Hongxing Li, Yixin Li, Dingming Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.