Memory heuristics fail LLM group decisions on CrossProfile tasks
Original title: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
On 480 crossed hidden-profile tasks, relevance readmits decoys and rarest-first is no better than random on average.
The paper introduces CrossProfile, a benchmark of 60 templates and 480 tasks that separates supporting, decoy, and irrelevant facts while holding their text fixed. Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, selecting memory by relevance or by rarity fails in predictable ways. Accuracy tracks the evidence balance the memory keeps. The decision model also overweight facts attributed to many agents, and full transcripts score worse than no notes.
Unconfirmed: Accuracy gaps are the authors' reported results on their benchmark.
- A perfect relevance filter loses 23 to 31 points to rarest-first selection.
- Rarest-first wins only when the rare facts are the useful ones.
- Dropping or randomly reassigning attribution tags raises accuracy by 13 to 24 points.
From the original article
1 min read at arxiv.orgWhen an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps.
By Anouar Imel · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.