Continual-ARC scores lifelong skill acquisition on ARC-AGI streams
Original title: Continual-ARC: Measuring Skill Acquisition over a Working Lifetime
Harness persistence rarely beats a demonstration cache on memorization, and no harness method gains on unseen tasks, unlike weight updates.
Continual-ARC is a lifelong stream built from the three ARC-AGI editions in which tasks return after dormant gaps with fresh instances while new tasks arrive. Learners may buy demonstrations, submit answers or act in a harness, and each instance is scored by the share of the feedback budget consumed. After the stream, frozen learners are tested on fresh instances of seen tasks and on unseen tasks. On Qwen3.5-9B and GLM-5.3 Flash, harness persistence generally does not improve memorization without explicit expert guidance, and most harness methods learn less than a cache of each task's demonstrations. No harness method gains on unseen tasks, while weight-based learning shows substantial gains.
- Tasks are drawn from all three ARC-AGI editions and need no domain knowledge.
- Models tested include Qwen3.5-9B and GLM-5.3 Flash, plus a Qwen3.5-9B trained to imitate GLM-5.3 Flash.
- Retries on incorrect submissions are allowed and charged against the feedback budget.
- Weight-based learning shows substantial gains on unseen tasks; harness methods do not.
From the original article
1 min read at arxiv.orgAgent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale.
By Daniel Lenton · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.