SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
SCOPD self-distillation helps vision-language models retain accuracy when most visual tokens are pruned.
SCOPD is a sparse-context on-policy self-distillation method in which a student reasons from pruned visual tokens while a full-context teacher supervises the same prefixes. It requires no ground-truth responses, architectural changes, or extra inference-time compute. At 10% visual-token retention, a vanilla model keeps 86.37% of unpruned performance across 13 benchmarks; SCOPD reaches 90.49% and SCOPD+ 92.43%. The authors attribute much of the loss to unreliable use of surviving evidence, not only irreversible information loss.
- Pass@K sampling shows pruned tokens still hold recoverable visual evidence
- SCOPD needs no labels, architecture changes, or extra inference compute
- SCOPD+ selectively distills visually sensitive response positions
- At 10% retention, retained performance rises from 86.37% to 92.43%
From the original article
1 min read at huggingface.co
Reasoning vision-language models (VLMs) process images and videos as long sequences of visual tokens, making inference expensive. Training-free token pruning reduces this cost, but aggressive compression can sharply degrade performance, often attributed to irreversible loss of task-relevant visual information. We show that this explanation is incomplete.
By Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.