Making Agents More Consistent: Skills Should Form Habits for Repeat Tasks
Paper proposes skill habit formation so agents cache deterministic skills from execution history, fixing 38-74% answer inconsistency and cutting tokens 14-56%.
Running 42 agent tasks three times each yielded non-agreeing answers in 38% to 74% of cases depending on model, with 95.3-97.2% of generated tokens spent re-deriving already-known plans. The authors propose 'skill habit formation': mining execution history for deterministic skill variants admitted through four cost-ascending gates, each claiming a region of input space. A habit-formed text-to-SQL variant reproduced all 456 repeated dispatches, was non-inferior in accuracy (p<0.0001), and used 14-56% fewer tokens, breaking even after 7-53 reuses. Trade-off: the guard admitted incorrect work on 2.6% of paraphrases and 26% of boundary inputs, and deterministic errors repeat exactly.
- 38-74% of repeated agent runs returned inconsistent answers
- Habit-formed variant reproduced all 456 repeated text-to-SQL dispatches
- Token use dropped 14-56%, net positive after 7-53 reuses
- Deterministic errors repeat exactly, making bad habits as reliable as good ones
Full article295 words · extracted from arxiv.org · click to collapse
On repeated work, agents are inconsistent. We ran 42 tasks three times each and found that, depending on the model, 38% to 74% returned answers that did not agree. Consistency is what a buyer, an auditor, or a regulator requires, and agents do not have it. They are wasteful too: 95.3% to 97.2% of what an agent generates goes to re-deriving a plan the system already knows. We propose skill habit formation. An agent mines its own execution history for candidate skills, deterministic variants that compete against the incumbent rather than replacing it. A candidate declares the region of input space it claims, so the common case runs as a script and the rest falls through to reasoning. Four gates of ascending cost admit candidates; the central one tests a candidate's execution trace against a retained reference, within a tolerance measured from that reference's own run-to-run variability. On text-to-SQL, three of four reasoning arms reproduced their own output on 11 to 13 of 42 repeated questions and the fourth on 26 of 42, while a habit-formed variant reproduced on all 456 dispatches we repeated and was non-inferior to every arm it replaced (p<0.0001). It also used 14% to 56% fewer tokens, turning net positive after 7 to 53 reuses. We measured what this costs in accuracy. The guard admitted work it should have deferred on 2.6% of natural paraphrases and 26% of inputs near its boundary, and 11 of 13 such failures were invisible to the trace-conformance gate at any threshold. Deterministic errors repeat exactly: a bad habit is as reliable as a good one, and that is the price of the property that makes the system auditable. Separating routing from parameter extraction raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.25299