Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs
Temporal Activation Injection restores fading mid-layer timing signals in VideoLLMs without training.
The authors find that reversing a video's frame order, which should invert temporal answers, often leaves VideoLLM predictions unchanged. A layer-wise temporal divergence signal peaks at intermediate layers and then fades toward the output, showing that timing information is learned but not preserved. Temporal Activation Injection extracts that peak for each input and reinjects it into later layers following the measured decay, with no training. It improved temporal reasoning across three VideoLLMs and four benchmarks while barely affecting non-temporal tasks.
- Reversed video frames often leave VideoLLM answers unchanged.
- Temporal divergence peaks mid-network and fades before the output.
- TAI reinjects that peak signal into later layers at inference.
- Gains hold on three VideoLLMs and four benchmarks without training.
From the original article
1 min read at huggingface.co
Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged.
By Youngwoo Shin, Yusung Ro, Minseo Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.