RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations
RealCompanion benchmarks whether agents truly use months of real companion chats to understand a person.
RealCompanion releases ten real long-term relationships with an AI companion: 27,218 messages spanning up to 120 days, plus a profile, persona, chat ground truth, and question set, each citing supporting messages. The authors report that a recency window locates the required message for 95.9% of probes and for only 2.2% of probes that actually need memory, and that 96% of the gain from supplying recorded evidence comes from messages that need none. Detectors failed to tell when memory is required on real messages, while labeling messages as memories raised their use by 10 to 14 points. Three agent systems reconstructed the persona at the same F1 despite a 31-fold cost difference.
- Ten real companion relationships total 27,218 messages over up to 120 days.
- A recency window finds the needed message for 95.9% of probes.
- Only 2.2% of probes that need memory are found that way.
- No tested detector reliably flags when memory is required.
- Three agent systems match persona F1 at a 31-fold cost gap.
Full article233 words · extracted from huggingface.co · click to collapse
A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing this requires a real person's record, and such records are private, so benchmarks generate the person and the questions and settle in advance what matters. We release \bench, ten real relationships with an AI companion: 27,218 messages over up to 120 days, released as the conversation and four files derived from it, a profile, a persona, a chat ground truth and a question set, each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation. Three findings follow. First, the past is rarely needed and far away. Pooled measures mislead: a recency window finds the required message for 95.9\% of probes and 2.2\% of those that need memory, and at the natural rate 96\% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector we tried can tell when memory is needed on real messages, authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.01780