Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
Mid-Harness samples and verifies terminal-agent actions before execution, lifting Pass@1 on TerminalBench-Lite.
Mid-Harness allocates test-time compute at the model-harness boundary by sampling candidate actions and verifying one before execution, without changing the generator or harness. Using TMAX-9B, weak verification yields little gain, but a GPT-5.6 Sol verifier raises TerminalBench-Lite Pass@1 from 50.00% to 68.03% with eight sampled actions. Pairwise verification works best when TMAX-9B itself is the verifier, and distilling the stronger verifier further improves Pass@1. Combining action and trajectory scaling beats generating more trajectories alone at lower estimated token cost.
- GPT-5.6 Sol verifier lifts Pass@1 from 50% to 68.03%.
- Eight sampled actions; generator and harness stay unchanged.
- TMAX-9B pairwise verification is the best self-verifier tested.
- Action-plus-trajectory scaling is cheaper than extra trajectories.
Full article223 words · extracted from huggingface.co · click to collapse
Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.39982