Memory heuristics fail LLM group decisions on CrossProfile
Original title: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
On 480 hidden-profile tasks, relevance and rarest-first memory selection both lose accuracy in predictable ways.
CrossProfile is a 60-template diagnostic with verified keys and 480 tasks that crosses ownership of supporting, decoy, and irrelevant facts while holding text fixed. Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, relevance filters readmit on-topic decoys and rarest-first selection is no better than random on average. Accuracy tracks the evidence the memory keeps; attribution tags add a social-proof bias, and full transcripts score below having no notes.
Unconfirmed: Accuracy figures are the authors' reported results and are not independently verified.
- A perfect relevance filter loses 23 to 31 points to rarest-first in the classic setup.
- Removing or randomly reassigning attribution tags raises accuracy by 13 to 24 points.
- Rarest-first wins only when the rare facts happen to be the useful ones.
- The complete transcript yields lower accuracy than no notes at all.
From the original article
1 min read at huggingface.co
When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps.
By Anouar Imel, Imad Charradi, Abderrahman Sghir · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.