Continual Learning Exam benchmarks long-horizon learning in LLM agents
Original title: Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Feedback plus one revision lifts median completion by 5.3 points, but no tested frontier setup finishes most five-level tasks.
Researchers introduce Continual Learning Exam (CLE), built with more than 50 domain experts, to test whether LLM agents improve from accumulated experience. It has 120 human-authored tasks in five-level ladders, 600 levels, 89 roles and 12 occupational domains. Agents must pass a level to unlock the next. The authors say the set can also be used to post-train models for continual learning.
Unconfirmed: Completion figures are the authors' reported results on their own benchmark.
- An evaluation agent scores submissions against human-authored rubrics.
- One setting allows diagnostic feedback and a single revision after failure.
- The comparison covers 17 harness-model configurations using frontier LLMs.
- The best system completes every level of a task in 23.3% of cases.
From the original article
1 min read at huggingface.co
Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience.
By Haoyu Zhao, Sitian Tu, Yuxuan Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.