Two papers on coding-agent benchmarks and critics

Separate papers urge task-flow benchmarks for coding agents and report Opera lifting resolve rates by up to 15 points.

Hugging Face daily papers on 2026-10-06 and 2026-10-07 carried two distinct coding-agent studies, not conflicting accounts of one result. The first argues benchmarks should match measured user task flows rather than issue tasks alone. From 4,782 production JetBrains IDE sessions, the authors focus on those with at least three user messages, about 33% of the sample, which mix questions, planning, review, refactoring, and execution. Their SWE-TaskFlow method preserves verified tasks and tests while steering interaction through prompt splitting and repository QA, scored by a TaskFlow Alignment Score; on 700 SWE-Bench Pro tasks, multi-step sequential solving roughly doubled cost without a stable resolve-rate change. The next day's paper introduces Opera, a persistent verbal critic that logs corrections as notes and follows them until the diagnosed problem is resolved. Across four policy models it lifts non-critic resolve rates by up to 12.4, 15.0, and 8.9 points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, and fine-tuning Qwen3.5-9B on Opera rollouts adds 10.2 points on held-out SWE-Bench Pro without a critic at inference, including after a harness switch from OpenHands to Terminus-2.