Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution
Video-RSI lets video agents rewrite their own harnesses, improving accuracy while inspecting fewer frames.
Video-RSI is a recursive self-improvement framework in which a video-understanding agent uses its own language model to revise the executable harness that controls what it observes. Because existing traces only contain evidence gathered by the current harness, the model revisits original training videos to test competing explanations for failures. Cost-aware evolution converts diagnoses into reusable harness revisions and retains those that improve answer accuracy relative to visual cost. Across reported video-understanding benchmarks, the evolved agent raises accuracy while processing fewer frames and remains competitive on the accuracy-efficiency trade-off.
- Video-RSI lets a video agent revise its own observation harness.
- The model reinvestigates training videos to test competing failure explanations.
- Cost-aware evolution keeps revisions based on accuracy and visual cost.
- Evolved agents improve accuracy while processing fewer frames.
From the original article
1 min read at huggingface.co
Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement.
By Bingjun Luo, Jialin Guo, Siqi Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.