FAPL trains MLLM latents to control cinematic video generation
Original title: Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
A diffusion transformer is grounded on frozen MLLM features, then the model is tuned for sparse prompts and single-factor edits.
Researchers propose Factor-Aware Predictive Learning so multimodal large language model latents steer a Video Diffusion Transformer instead of being bypassed. The method first trains the generator on a frozen MLLM and connector, then uses paired videos for factor-span edits, and finally freezes the transformer while optimizing the MLLM for limited inputs. They report better results than baselines on sparse-to-video, factor-aware editing, and multi-shot generation.
- Latent-only grounding trains the diffusion transformer while the MLLM and connector stay frozen.
- Paired videos specify one target-factor change while other factors are preserved.
- A later stage freezes the transformer and updates the MLLM for sparse observations.
- Reported tasks are sparse-to-video, factor-aware editing, and multi-shot generation.
From the original article
1 min read at huggingface.co
Cinematic video generation requires both temporal inference from limited inputs (i. e. , sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera.
By Songlin Yang, Zhe Wang, Xiaotong Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.