Streaming Video Editing with Easy Adaptation
SVEET adapts a bidirectional video diffusion model for real-time streaming edits at 15 FPS on one H100.
SVEET adapts a pretrained bidirectional video diffusion model for autoregressive streaming video editing. An auxiliary branch encodes source frames with temporally independent self-attention and injects those features into backbone blocks, while decoupled training keeps controllability orthogonal to causality. The authors report superior editing quality and 15 FPS on a single H100 without auxiliary acceleration, and they released the code.
- Identifies feature disentanglement and conditional frame independence.
- Auxiliary branch injects temporally independent source features into backbone blocks.
- Decoupled training separates controllability optimization from model causality.
- Reports 15 FPS streaming edits on one H100 without extra acceleration.
Full article186 words · extracted from huggingface.co · click to collapse
In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.24788