Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency
LoHi pairs dense low-resolution video with sparse high-resolution frames, raising long-video accuracy and cutting decode latency.
Researchers show that for long-video vision-language models, dense low-resolution sampling beats sparse native-resolution sampling at the same token budget, while front-end decoding dominates wall time on hour-long videos. They introduce LoHi, a training-free single-pass method that mixes a dense low-resolution video stream with sparse high-resolution frames through the model's native video and image pathways. LoHi-Anchor selects frames from codec I-frames, and LoHi-SemDiv uses query relevance plus CLIP visual diversity. Across three benchmarks, LoHi gains 10.6 points over the native-resolution baseline and 5.2 points over the strongest prior efficiency method, cutting decode latency by up to 7x.
- Dense low-resolution sampling beats sparse native resolution at matched token budgets.
- LoHi is training-free and uses the model's native video and image pathways.
- Accuracy rises 10.6 points over native resolution and 5.2 over the prior best.
- Front-end decoding latency falls by up to 7x on hour-long videos.
From the original article
1 min read at huggingface.co
Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget.
By Sixun Dong, Wei Li, Andong Deng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.