Opera: A Verbal Critic Framework for Long-horizon Coding Agents
Opera, a persistent verbal critic, lifts coding-agent resolve rates by up to 15 points on SWE-Bench-style benchmarks.
Opera is a test-time verbal critic that stores each correction as a persistent note and follows it until the diagnosed coding problem is actually resolved, using periodic and event-driven triggers, typed operators, and an evidence audit before delivery. Across four policy models it raises non-critic resolve rates by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, and leads competing critic baselines. Fine-tuning Qwen3.5-9B on Opera-guided rollouts improves held-out SWE-Bench Pro resolve rate by 10.2 points without a critic at inference, matching stronger-model rollouts and remaining stable when the harness changes from OpenHands to Terminus-2.
- Corrections are persistent notes tracked until the diagnosed problem is resolved.
- Resolve rate rises by up to 15.0 points on a SWE-Bench Pro subset.
- Fine-tuning Qwen3.5-9B on Opera rollouts adds 10.2 points without a critic.
- Gains hold when switching harness from OpenHands to Terminus-2.
Full article225 words · extracted from huggingface.co · click to collapse
Long-horizon coding agents need timely corrections, yet feedback can be ineffective or even harmful when it misjudges ongoing work or fails to address the underlying problem. Existing critics focus on evaluating trajectories and generating feedback, but rarely track what happens after feedback is delivered. We present Opera, a verbal critic framework that treats each correction as a persistent note, followed until the diagnosed problem is resolved. Opera decides when to review through periodic and event-driven triggers, diagnoses issues with typed operators, audits feedback against visible evidence before delivery, and tracks the agent's subsequent actions to distinguish mere compliance from actual resolution. As a test-time critic, Opera improves the resolve rate of non-critic agents by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, respectively, across four policy models, and achieves the highest mean resolve rate among competitive critic baselines on all three benchmarks, and also improves policy models when the policy critiques itself. Beyond inference, Opera-guided rollouts provide approximately on-policy training data: fine-tuning Qwen3.5-9B on them improves its resolve rate on held-out SWE-Bench Pro repositories by 10.2 percentage points without a critic at inference time, matching fine-tuning on rollouts from a stronger model, while preserving its performance when switching harness, i.e., from Openhands to Terminus-2, which the latter substantially degrades. Our code is available at: https://github.com/dongyuanjushi/Opera.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.33987