Memory as Weights: Internalizing Long-Term History for Streaming Videos
MaLoW stores streaming video history as LoRA weights, beating prior memory methods on OVO-Bench.
MaLoW is a streaming vision-language memory framework that uses a recurrent HyperNetwork to fold past video into evolving LoRA weights, so history does not compete with the current frame for context. The authors also build MaLoW-Instruct, with 381K question-answer pairs over 56K videos. On the OVO-Bench backward split it outperforms prior memory methods by 3.9% to 11.8% across video backbones with modest latency, and it improves action success on Simpler WidowX by 9.3%.
- Recurrent HyperNetwork consolidates video into fixed-size LoRA weights.
- MaLoW-Instruct contains 381K QA pairs over 56K videos.
- Beats prior memory methods by 3.9% to 11.8% on OVO-Bench backward.
- Raises Simpler WidowX action success rate by 9.3%.
- Products
- MaLoW-Instruct
- AI models
- MaLoW
From the original article
1 min read at huggingface.co
Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and user queries. Existing memory mechanisms typically compress, retrieve, or cache historical observations and reintroduce them into the input context, forcing historical memory and current observations to compete for limited context.
By Zhuoming Liu, Jinhong Lin, Lin Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.