LLMs still leak real-item knowledge in low-knowledge persona simulations
Across five API endpoints from two model families, explicit ignorance prompts still left a favourable-item real-twin rating gap.
Stories per day · 30 days
Relatedbenchmark2benchmarks1coding-agents1e-commerce1knowledge-leakage1llm-agents1llm-evaluation1multi-turn1persona-simulation1qwen31rating-bias1recommendation-agents1swe-bench1tau-bench1tool-calling1trajectory-distillation1
6 stories, newest first · one card per story
Stories tagged #user-simulation
UserArena benchmarks shopping agents with 64,729 real session trajectories
On-Policy Trajectory Distillation is reported to best match a production A/B policy effect among tested simulators.
SWE-DIVE evaluates coding agents in multi-turn sessions
A user simulator and conversation validator run inside Harbor to score whether the agent solves the task.
τ-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions
τ-Emotion-Bench shows emotional users expose reliability gaps in tool-calling LLM agents.
Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States
Mind2Dialogue simulates users' mental states to create privileged supervision, boosting personalization metrics of Qwen, Llama, and OLMo assistants by up to 40.9 points.
Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States
Mind2Dialogue simulates users' mental states to generate privileged supervision, boosting personalization and preference-following in Qwen, Llama, and OLMo assistants.