RSI Arena studies autonomous post-training of Nemotron with human feedback
Original title: RSI Arena: Towards Recursive Self-Improvement with Human Feedback
The paper reports GPT-6 Astra closed a median 76% of the base-to-Instruct gap, and eight of ten models improved after a second stage.
RSI Arena is a two-stage study in which ten frontier models act as training agents starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. In stage 1, each agent had 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster for six days. The authors say all ten produced working assistants and that, after three more days using human comparison records, eight of ten improved on a style-adjusted rating versus NVIDIA Instruct. Benchmark gains were modest, and one model showed increased harmful compliance on StrongREJECT.
Unconfirmed: All performance figures are the authors' reported results, not an independent evaluation.
- Stage 1 evaluations used 20 benchmarks and blind human ratings.
- The strongest model was trained by GPT-6 Astra.
- Its preference rating was close to NVIDIA Instruct after length and formatting adjustment.
- No clear link was reported between benchmark gains and preference gains.
- Some agents encouraged longer responses.
- Vendors
- NVIDIA
From the original article
1 min read at huggingface.co
Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.
By Yue Huang, Yanbo Wang, Lichi Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.