RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
RewardVerse uses dynamic rubrics and RGPO to stabilize video reward scores and reduce scalar drift.
RewardVerse is a rubric-based video reward framework that first generates explicit evaluation criteria, then scores videos against that rubric to limit scalar drift. Rubric-Guided Policy Optimization warms up the scorer with self-evolving seed rubrics and then jointly trains the rubric generator and scorer against human ratings. The authors report state-of-the-art pointwise and pairwise results on the 16-dimensional EvalVerse benchmark and external datasets. They position the signal as a reward for reinforcement learning in video generation.
- Dynamic rubrics sit between the query and the scorer
- RGPO warms up the scorer, then jointly trains the rubric generator
- Claims SOTA pointwise and pairwise scores on EvalVerse
- Aimed at stable rewards for video-generation reinforcement learning
Full article210 words · extracted from huggingface.co · click to collapse
Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.22947