Skip to content
NewA calmer, faster ZeroHour

Spool speeds hybrid VLM streaming inference without replaying expired frames

Original title: Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

arXiv cs.AI / cs.LG / cs.CLby Rui Ning ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

A training-free cache cuts p95 time-to-first-token by a geometric mean of 6.6× versus the faster of two vLLM baselines.

Spool is a training-free framework for hybrid vision-language models that answer questions over a sliding window of recent video frames. Recurrent layers normally force a full replay when old frames expire; Spool caches per-frame state transformations and composes them as the window advances. Across five open-source hybrid VLMs and four video reasoning benchmarks, it had the lowest p95 time-to-first-token in all 20 tests and raised accuracy by 2.3 percentage points on average. Removal is exact within each layer, though earlier layers can leave residual influence.

Key facts
  • The paper compares Spool with two recent vLLM baselines.
  • It reports a 6.6× geometric-mean cut in p95 time-to-first-token.
  • Average accuracy rose by 2.3 percentage points.
  • Expired frames can still affect cached inputs from earlier layers.
  • The method needs no extra training and uses custom GPU kernels.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e. g. , the last two minutes).

arXiv cs.AI / cs.LG / cs.CL

By Rui Ning · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.