SWE-DIVE evaluates coding agents in multi-turn sessions
Original title: SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
A user simulator and conversation validator run inside Harbor to score whether the agent solves the task.
The paper argues that one-shot benchmarks such as SWE-bench miss poorly specified, multi-turn coding-agent work. SWE-DIVE adds a user simulator with different behavioral patterns and a validator that checks whether simulator messages meet the original requirements and steer the agent toward a solution. Harbor orchestrates the agent, simulator and validator and runs the test environment. Results and source code are linked from the authors' GitHub repository.
- Existing GitHub issues are treated as too well formulated for real use.
- The simulator can vary user behavior across turns.
- The validator checks simulator outputs against the initial requirements.
- Code is at github.com/lorenzbrehme/swe-dive_usersim.
From the original article
1 min read at arxiv.orgCoding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues.
By Lorenz Brehme · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.