OmniXRef unifies text, image, and video references for editing
Original title: OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing
A diffusion transformer jointly edits a source video from variable text, image, and video references, and the authors say it beats commercial models.
Researchers introduce OmniXRef, a framework for combining variable numbers of text, image, and video references to control attributes of a source video. OmniXRef-Model uses a single-stream diffusion transformer with interleaved MLLM context and R^2-RoPE for additional references. They built OmniXRef-Data with synthesis pipelines and curriculum learning, plus OmniXRef-Bench for Camera, Motion, Content, Style, and VFX transfer. The paper reports better results than task-specific baselines and commercial models.
Unconfirmed: Outperformance claims are the authors' evaluation results, not independently verified.
- A reference-bank interface accepts a variable number of multimodal references.
- R^2-RoPE is introduced so the model can generalize to more references.
- OmniXRef-Bench covers five video-reference attribute-transfer categories.
- Synthetic pipelines and curriculum learning address scarce video-reference data.
From the original article
1 min read at huggingface.co
Creative intent in video editing can be expressed through text instructions (e. g. , "remove the people"), image references (e.
By Xiaoyan Cong, Tianyu Wang, Hyeonho Jeong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.