LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation
LIFT controls future-frame layout in image-to-video generation under large camera viewpoint changes.
LIFT is an image-to-video framework that adds last-frame layout control to camera control so users can specify what appears in a future view and where. On-policy self-distillation transfers control from a dense per-frame layout teacher to a student conditioned only on the last-frame layout, which is harder under large viewpoint changes that reveal unseen regions. The authors curate LIFT-Vista with large viewpoint changes and temporally consistent camera and layout annotations. Experiments report better video quality, future-layout controllability, and camera controllability than prior methods.
- Users specify last-frame content and layout plus camera motion.
- On-policy self-distillation transfers dense-layout control to a sparse student.
- LIFT-Vista provides large viewpoint changes with consistent layout labels.
- Reported gains cover quality, future-layout control, and camera control.
- AI models
- LIFT
From the original article
1 min read at huggingface.co
We introduce LIFT, a unified image-to-video generation framework that complements camera control with Layout-In-FuTure control, enabling users to specify what should appear in a future view and where it should appear.
By Shengxiang Ji, Boyang Wang, Haiyang Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.