Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
Learn2Play Bench tests whether LLM agents learn novel game rules from interaction instead of pretrained knowledge.
Learn2Play Bench is a suite of newly designed text-based games whose rules are novel or counterintuitive, so agents must learn from interaction rather than pretrained knowledge. Games give reproducible feedback and automatic scores, and instances vary to test transfer. The authors find that keeping full action-and-feedback records beats summarizing experience into rules, that top humans still outscore agents, and that changing the harness can raise performance while lowering estimated inference cost.
- Games use novel or counterintuitive rules unknown to pretrained models.
- Full action-and-feedback logs beat summaries of rules or strategies.
- Humans still outscore agents and explore more varied strategies.
- With a fixed backbone, harness choice raises scores and cuts inference cost.
- Products
- Learn2Play Bench
From the original article
1 min read at huggingface.co
Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge.
By Yibo Li, Jinhang Qiu, Zhi Zheng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.