HABIT-Bench finds agent memory weak at inferring user habits
Original title: HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory
The best of eight memory methods scored 33.61% macro accuracy on Qwen3-235B-A22B and GLM-5.3-Flash across four domains.
HABIT-Bench tests whether LLM agent memory can infer habits from weak evidence in long synthetic user histories and apply them to later requests. Histories span hundreds of sessions in food, finance, software, and travel, built from task-oriented dialogue and controlled habit graphs. Four-choice tasks include temporary exceptions, habit changes, and suggestions users never accepted; sampled tasks received independent human review. Performance stayed limited even when annotated evidence was supplied directly.
- Eight memory methods were evaluated with Qwen3-235B-A22B and GLM-5.3-Flash.
- The best method reached 33.61% macro accuracy across four domains.
- Tasks include exceptions, habit changes, and unaccepted suggestions.
- Giving annotated evidence directly still left accuracy limited.
From the original article
1 min read at huggingface.co
In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies.
By Junming Zhang, Rubin Wei, Xu Cheng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.