Continual harness evolution causes forgetting in coding agents
Original title: Can Coding Agents Sustain Continual Evolution without Forgetting?
Meta-Harness drops 66% on SWE-Bench after evolution on FeatureBench with DeepSeek-V4-Flash-0731.
Agent harness self-evolution improves performance on single tasks but lacks evaluation for multiple tasks. Experiments across six evolution directions on software engineering datasets show subsequent evolution undermines prior performance. With DeepSeek-V4-Flash-0731, Meta-Harness drops from 94% to 32% on SWE-Bench after rounds on FeatureBench.
- Subsequent dataset evolution undermines prior performance
- Meta-Harness drops 66% relative on SWE-Bench after FeatureBench
- Three datasets organized into six directions
- Identifies insights for continual self-evolution
- AI models
- DeepSeek-V4-Flash-0731
From the original article
1 min read at huggingface.co
Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task.
By Shangziqi Zhao, Yitong Zhang, Jia Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.