Continual Learning Exam shows frontier agents stall on multi-level tasks
Original title: Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
The best of 17 setups cleared 47.7% of levels and finished all five levels on 23.3% of tasks.
Continual Learning Exam measures whether LLM agents improve across experience in professional workflows. It has 120 human-authored tasks, each a five-level ladder, for 600 levels across 89 roles and 12 occupational domains. Agents must pass a level to unlock the next, and an evaluation agent scores work against human rubrics. Feedback plus one revision raised median completion by 5.3 percentage points versus a single submission.
- More than 50 domain experts authored the tasks and rubrics.
- Later levels build on experience from earlier assignments.
- Seventeen harness-model configurations of frontier LLMs were compared.
- The authors say the dataset can also be used to post-train continual learning.
- Products
- Continual Learning Exam
From the original article
1 min read at arxiv.orgContinual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience.
By Haoyu Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.