LVMT: Video Mask Transformer for Long-term Video Segmentation
LVMT uses GRU memory and truncated query propagation to set a faster state of the art in long-term video segmentation.
LVMT is a Long-term Video Mask Transformer for online video segmentation that uses a lightweight GRU to choose which object information to keep and propagate across time. Truncated Query Propagation trains on video chunks so object state crosses chunks while backpropagation stays inside each chunk, avoiding memory limits and vanishing gradients. On six benchmarks the authors report a new state of the art while remaining about 10 times faster than the previous best.
- A GRU memory module selects which object information to propagate over time.
- Truncated Query Propagation trains on long videos without full backpropagation.
- LVMT claims new state of the art on six video segmentation benchmarks.
- Authors say it stays as fast as its base model and is 10x faster than prior SOTA.
From the original article
1 min read at huggingface.co
Existing online video segmentation methods struggle to track objects in long, complex videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long videos due to memory requirements and vanishing gradients.
By Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.