CARTBench: shopping models lag on excellence criteria
Original title: CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines
Three of 10 models pass 90% on Core, but none exceeds 45% on Excellence across 1,328 criteria.
CARTBench evaluates multi-turn shopping advice for complex, ambiguous requests with timelines. Cases come from real commercial needs in 17 retail domains, with timestamped dialogue and 1,603 product records. Case-specific rubrics follow Ideal Response Principles taken from strong replies on completed orders, and the same principles define Principle-Guided CART Prompting. The first full evaluation under frozen rubrics covers 7 flagship APIs, including GPT-6.1-Sol and Claude-Opus-5, plus open-weight models above 35B parameters. The authors report an understanding-to-execution gap and shared failures on product evidence and unresolved needs.
Unconfirmed: Model names beyond GPT-6.1-Sol and Claude-Opus-5 are not listed, and scores are the authors' first frozen-rubric run.
- The suite has 1,328 expert-annotated Core, Excellence and Safety criteria.
- No evaluated model exceeds 45% on Excellence.
- Failures tie to changing requirements and decisive unknowns.
- Rubrics were distilled from manually selected successful-order replies.
- Code and data are at Da1yuqin/CARTBench.
From the original article
1 min read at huggingface.coLarge language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines).
By Yuqin Dai, Zhanwei Zhang, Honghui Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.