FrameForge Uses Video Model Priors for Unified Image Generation
Original title: FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation
FrameForge models condition-target relationships with Mixture of Temporal Experts and Chain of Temporal Reasoning for superior subject-driven generation.
Unified conditional image generation requires preserving aspects of visual input while following textual guidance. Pretrained video models encode visual transformations across time but typically assign conditions and targets to fixed positions. The paper observes that condition-target relationships vary across tasks and video models encode transformations at different temporal scales. FrameForge captures these via Mixture of Temporal Experts selecting dynamic temporal RoPE experts and Chain of Temporal Reasoning with a reasoning-bottleneck attention mask.
- Mixture of Temporal Experts models target distance from condition
- Chain of Temporal Reasoning models transformation via latent intermediate state
- Superior performance on DreamBench, COCO2017, and GEdit-Bench
- Effective for subject-driven generation, canny-to-image, and image editing
- Uses input-adaptive virtual temporal relations
From the original article
1 min read at huggingface.co
Unified conditional image generation requires a single model to preserve different aspects of a visual input while producing a target image that follows textual guidance. Pretrained video models provide a promising foundation by encoding visual transformations across time, yet existing approaches typically assign the condition and target to fixed temporal positions.
By Yijing Lin, Zhendong Mao, Mengqi Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.