Scalable Weak-to-Strong Verifier Hardening for Agent Environments
A hacker-fixer loop automatically hardens agent verifiers on Terminal Bench, KernelBench, and SETA against reward hacking.
The paper proposes fully automatic weak-to-strong hardening of programmatic verifiers used to score LLM agents. An isolated sandbox blocks direct verifier edits, then a hacker-fixer loop searches for reward-hacking exploits and repairs them. Applied to Terminal Bench, KernelBench, and SETA, the loop substantially cuts attack success and strengthens defenses as iterations increase, beating a static baseline. An earlier version of the loop was adopted by the official Terminal Bench repository to audit environment exploits.
- Isolated sandbox plus a hacker-fixer loop finds and patches verifier bugs.
- Attack success drops on Terminal Bench, KernelBench, and SETA.
- More loop iterations beat a static non-iterative baseline.
- An earlier version was adopted by the official Terminal Bench repo.
Full article190 words · extracted from huggingface.co · click to collapse
Agent environments are essential for large language model (LLM) development. They use programmatic verifiers to turn complex outcomes into rewards. Verifiers can be attacked into awarding full credit to cheating agents without completing the intended task, causing inaccurate evaluation and misalignment in training. As agent environments scale in numbers and difficulty, so must the repairing of exploitable verifiers. To this end, we ask whether today's model with scaled compute can automatically harden agent environments to prevent potentially stronger future models from reward hacking. We introduce a scalable, fully automatic approach to weak-to-strong verifier hardening. It first sets up an isolated verifier sandbox that agents cannot directly modify, then uses a hacker-fixer loop to find and repair remaining verifier vulnerabilities. We apply this loop to Terminal Bench, KernelBench, and SETA, substantially reducing attack success across these agent environments. The iterative design scales with the hardening budget, producing progressively stronger defenses as the number of loop iterations increases and outperforming the static, non-iterative baseline. An earlier version of our hacker-fixer loop has been adopted by the official Terminal Bench repository as a way to audit exploits in its environments. Project page: https://verifier-hardening.github.io/.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.15924