UserArena benchmarks shopping agents with 64,729 real session trajectories
Original title: UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
On-Policy Trajectory Distillation is reported to best match a production A/B policy effect among tested simulators.
UserArena is an e-commerce benchmark for interactive user simulation of long-horizon shopping agents. It contains 64,729 session trajectories from 11,010 users, covering 566,102 items and 13 action types, plus an interactive environment and offline checks against a controlled production A/B test. Evaluation has five levels from output validity and local decisions through sequential behavior, personalization, and outcomes. The authors propose On-Policy Trajectory Distillation, which uses trajectory-level intent as dense teacher targets on student rollouts.
Unconfirmed: Fidelity and A/B reproduction claims are the authors' results under their protocol.
- Trajectories come from 11,010 users and span 566,102 items.
- The data includes 13 action types and supports trajectory-level evaluation.
- Offline validation compares simulator outcomes with a production A/B test.
- OPTD is reported to have the strongest overall fidelity under the protocol.
From the original article
1 min read at huggingface.co
Agentic recommendation pursues shopper goals through sequences of context-dependent decisions. Developing and evaluating such policies requires faithful recommendation world models that capture both the interactive shopping environment and users' behavioral responses over long interactions.
By Jujia Zhao, Jiakai Tang, Jing Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.