ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
ShallowStream builds streaming-video retrieval indexes from shallow MLLM layers, cutting per-frame prefill latency by up to 52.1x.
ShallowStream is a framework for streaming video understanding with multimodal LLMs that uses the model's shallow layers to simultaneously encode frames and maintain an always-on lightweight retrieval index via shallow-layer KV caches, avoiding full-depth prefill for every incoming frame. At query time, shallow-layer attention scores plus a diversity-aware selection strategy retrieve relevant context frames. It reports performance on par with the strongest existing streaming methods while reducing per-frame prefill latency by up to 52.1x and 10-second end-to-end latency by up to 11.9x, with code released on GitHub.
- Uses shallow-layer KV caches as an always-on index instead of full-depth prefill per frame.
- Matches strongest streaming baselines while cutting per-frame and 10-second end-to-end latency up to 52.1x and 11.9x.
From the original article
1 min read at huggingface.co
Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive.
By Jitai Hao, Ke Yang, Qiang Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.