Switch-SWE benchmark shows coding agents lose focus on interrupted tasks
Original title: Switch-SWE: Do Coding Agents Hold Focus When Switching Between Unfinished Tasks?
Interleaved unfinished tasks reduce focus-task resolution by 11.5 percentage points across evaluated coding agents.
Switch-SWE is a new benchmark for testing coding agents' ability to resume focus tasks after switching to other unfinished work in the same repository. The benchmark contains 200 focus tasks from 19 Python repositories rendered as 400 matched session specifications. Evaluation of 7 agents found an average 11.5 percentage point resolution gap when interleaving tasks, with cross-task confusion as a common failure mode. A plug-in context-management module improved focus-task completion by 5.5 to 15.5 percentage points.
- Switch-SWE uses 200 focus tasks from 19 Python repositories
- Task switching causes average 11.5 percentage point resolution drop
- Proposed module improves completion by 5.5-15.5 points
- Code available at github.com/Miraclecsc/Switch-SWE
From the original article
1 min read at huggingface.co
Software development sessions often involve several unfinished tasks rather than one issue carried through to completion. As coding agents take on repository-level development, they need to recover an earlier task's requirements and partial progress after intervening work.
By Shuochen Chang, Qianli Ma, Yibo Miao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.