Continual-ARC finds harness methods fail on unseen tasks
First merged summary.
On Continual-ARC, harness methods do not gain on unseen tasks, while weight-based learning shows substantial gains.
Continual-ARC is a lifelong stream built from all three ARC-AGI editions. Tasks return after dormant gaps with fresh instances while new tasks arrive. Learners may buy demonstrations, submit answers, or act in a harness, and each instance is scored by the share of the feedback budget used. On Qwen3.5-9B and GLM-5.3 Flash, harness persistence generally does not improve memorization without explicit expert guidance, and most harness methods learn less than a cache of each task's demonstrations. No harness method gains on unseen tasks, while weight-based learning shows substantial gains.
- Tasks come from all three ARC-AGI editions and need no domain knowledge.
- Models include Qwen3.5-9B, GLM-5.3 Flash, and a Qwen3.5-9B trained to imitate GLM-5.3 Flash.
- Incorrect submission retries are allowed and charged against the feedback budget.
- Memory-based and weight-based systems share one scoring scale.
- Frozen learners are tested on fresh seen-task instances and on unseen tasks.
What each outlet adds
beyond the shared facts- arXivAdds retry charging, imitation training, and no domain knowledge.
- Hugging Face daily papersAdds that memory and weight systems share one scoring scale.
- Products
- Continual-ARC
- AI models
- Qwen3.5-9BGLM-5.3 Flash
Coverage
oldest first · one row per article · UTCContinual-ARC scores lifelong skill acquisition on ARC-AGI streams
Harness persistence rarely beats a demonstration cache on memorization, and no harness method gains on unseen tasks, unlike weight updates.
Continual-ARC measures lifelong skill learning on repeated ARC-AGI tasks
Harness persistence rarely beat caching demonstrations; weight updates helped unseen tasks, harness methods did not.