Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
A Builder learns reusable meta-skills that improve frozen-model agent harnesses by about nine points on benchmarks.
The paper studies test-time AI-for-AI, where a Builder with frozen weights learns to construct better execution environments for a frozen Target model. Reusable Meta-Skill principles specify when support is needed and what resources to provide, learned from the Target's development-set feedback. On Harness-Bench and NewtonBench, a full skill bank improves macro-average performance by 8.95 percentage points over no-skill construction and by 12.02 points over handing that bank directly to the Target. Gains also appear when the same model fills both roles.
- A Builder learns principles for when a frozen Target needs support and which resources to supply.
- Skills come from development-set execution feedback, then a frozen bank builds harnesses for new tasks.
- Full-bank meta-skills improve macro-average performance by 8.95 points over no-skill construction.
- They also beat giving the same bank directly to the Target by 12.02 points.
Full article145 words · extracted from arxiv.org · click to collapse
Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.38143