Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
A survey catalogs inference-efficiency techniques for video and audiovisual LLMs, mapping bottlenecks in sampling, encoding, token reduction, and LLM decoding.
This survey covers inference-efficiency mechanisms for visual and audiovisual video LLMs, reporting reductions in parameters, FLOPs, latency, memory, and token counts. It organizes methods by pipeline stage, covering frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding for systems built since late 2022. The authors compile accuracy-cost comparisons under shared host models and input protocols, identify gaps in audiovisual efficiency and standardized evaluation, and maintain a public repository.
- Survey covers efficiency methods across frame sampling, modality encoding, token reduction, and decoding.
- Compiles accuracy-cost comparisons under shared host models and protocols.
- Identifies gaps in audiovisual efficiency and standardized evaluation; maintains a public repository.
Full article171 words · extracted from huggingface.co · click to collapse
Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.10355