Making Agents More Consistent: Skills Should Form Habits for Repeat Tasks
Paper proposes skill habit formation so agents cache deterministic skills from execution history, fixing 38-74% answer inconsistency and cutting tokens 14-56%.
Running 42 agent tasks three times each yielded non-agreeing answers in 38% to 74% of cases depending on model, with 95.3-97.2% of generated tokens spent re-deriving already-known plans. The authors propose 'skill habit formation': mining execution history for deterministic skill variants admitted through four cost-ascending gates, each claiming a region of input space. A habit-formed text-to-SQL variant reproduced all 456 repeated dispatches, was non-inferior in accuracy (p<0.0001), and used 14-56% fewer tokens, breaking even after 7-53 reuses. Trade-off: the guard admitted incorrect work on 2.6% of paraphrases and 26% of boundary inputs, and deterministic errors repeat exactly.
- 38-74% of repeated agent runs returned inconsistent answers
- Habit-formed variant reproduced all 456 repeated text-to-SQL dispatches
- Token use dropped 14-56%, net positive after 7-53 reuses
- Deterministic errors repeat exactly, making bad habits as reliable as good ones
From the original article
1 min read at arxiv.orgOn repeated work, agents are inconsistent. We ran 42 tasks three times each and found that, depending on the model, 38% to 74% returned answers that did not agree. Consistency is what a buyer, an auditor, or a regulator requires, and agents do not have it.
By Travis Weber · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.