Coding-Agent Benchmarks Should Match Their Users' Task Flows
Coding-agent benchmarks should be calibrated to real IDE session task flows, not issue tasks alone.
The paper argues coding-agent benchmarks should match measured user task flows rather than a single notion of realism. From 4,782 production sessions of software engineers in JetBrains IDEs, the authors focus on sessions with at least three user messages, about 33% of the sample, which span questions, planning, review, refactoring, and execution. SWE-TaskFlow preserves verified tasks and tests while steering interaction via prompt splitting and repository QA, scored by a TaskFlow Alignment Score. In a pilot on 700 SWE-Bench Pro tasks, solving sequentially in several steps roughly doubled agent cost without a stable change in resolve rate.
- Authors analyzed 4,782 real JetBrains IDE coding-agent sessions.
- Long sessions mix questions, planning, review, refactoring, and execution.
- SWE-TaskFlow steers issue benchmarks toward a measured Task Flow.
- On 700 SWE-Bench Pro tasks, multi-step solving about doubled cost.
- Products
- SWE-Bench ProSWE-TaskFlow
- Organizations
- JetBrains
From the original article
1 min read at huggingface.co
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample).
By Igor Slinko, Yaroslav Golubev, Sergey Titov · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.