FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
FastBench shows streaming VLMs miss fast events; Gemini-3.5-Flash scores only 50.7%.
FastBench evaluates whether streaming video language models can perceive fast real-world events under limited context. It has 306 trajectory-verified QA pairs spanning eight domains, six capabilities, and forward, instant, and backward temporal scopes. Gemini-3.5-Flash scores 50.7%, while denser sampling lifts Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS. A training-free ProactiveFrame baseline beats sparse uniform sampling by 5.4 and 1.5 points but remains far below oracle-guided focusing.
- FastBench contains 306 QA pairs across eight domains and six capabilities.
- Strongest model Gemini-3.5-Flash scores only 50.7%.
- Qwen3-VL-8B improves from 32.9% at 2 FPS to 44.6% at 24 FPS.
- ProactiveFrame beats sparse sampling by up to 5.4 points but trails oracle focusing.
Full article209 words · extracted from arxiv.org · click to collapse
Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.12427