CrossProfile finds LLM group memory still picks the wrong facts
First merged summary.
On 480 hidden-profile tasks, relevance and rarest-first memory both fail in predictable ways across four open models.
A paper introduces CrossProfile, a 60-template diagnostic with verified keys and 480 tasks that crosses ownership of supporting, decoy, and irrelevant facts while holding their text fixed. Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, relevance filters readmit on-topic decoys, and rarest-first selection is no better than random on average. Accuracy tracks the evidence the memory keeps. Attribution tags add a social-proof bias: the decision model overweights facts credited to many agents, and dropping or randomly reassigning those tags raises accuracy by 13 to 24 points. Full transcripts score worse than having no notes.
- CrossProfile has 60 templates and 480 crossed hidden-profile tasks.
- Models tested: Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models.
- A perfect relevance filter loses 23 to 31 points to rarest-first in the classic setup.
- Rarest-first wins only when the rare facts are the useful ones.
- Removing or randomly reassigning attribution tags raises accuracy by 13 to 24 points.
- Full transcripts score lower than having no notes at all.
Timeline
dated events in the reporting · oldest first- Oct 26, 2026Hugging Face daily papers lists the CrossProfile study. · Hugging Face daily papers
- Oct 27, 2026The paper is posted on arXiv. · arXiv
What each outlet adds
beyond the shared facts- Hugging Face daily papersClassic-setup 23–31 point gap and transcript-versus-no-notes result.
- arXivDecision model overweights facts attributed to many agents.
- AI models
- Qwen2.5-7BLlama-3.1-8B
Coverage
oldest first · one row per article · UTC- · 4d ago
Hugging Face daily papersFirst report
Memory heuristics fail LLM group decisions on CrossProfile
On 480 hidden-profile tasks, relevance and rarest-first memory selection both lose accuracy in predictable ways.
- · 3d ago
arXiv cs.AI / cs.LG / cs.CL
Memory heuristics fail LLM group decisions on CrossProfile tasks
On 480 crossed hidden-profile tasks, relevance readmits decoys and rarest-first is no better than random on average.