VideoGen-Agent: Reinforcing Video Generation Agents
VideoGen-Agent uses reinforcement learning and tools to lift text-to-video scores by 19.1 points on VABench.
VideoGen-Agent is a multimodal agent trained with multitask agentic reinforcement learning to coordinate augmentation, generation, and verification tools for video generation. Supervised fine-tuning on teacher trajectories is refined with a category-aware hybrid reward covering tool validity, appropriate tool use, and video quality. On the 600-prompt VABench, it improves from 56.5 to 75.6 over its base text-to-video generator, and to 86.1 when generation tools are upgraded without retraining. Human raters preferred the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons.
- Shared policy trained across six video-generation task categories.
- VABench holds out 600 prompts covering identity, physics, and multi-shot structure.
- Score rises 19.1 points, from 56.5 to 75.6, versus the base generator.
- Upgraded generation tools reach 86.1 without additional agent training.
- Human raters preferred the upgraded agent in 84.3% of comparisons.
Full article209 words · extracted from huggingface.co · click to collapse
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.24997