TeamGym benchmarks language-model teams on distributed decision tasks
Original title: TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information
Splitting pivotal facts across agents drops success by 22.9 points, and requiring retrieval drops it another 19.9.
TeamGym is an environment for evaluating and training language-model agents on team decisions built from legal, finance, medical, and software-engineering corpora. It defines pivotal information as a minimal set of facts that are each necessary and jointly sufficient for a correct decision, then controls how those facts are distributed. Across 8 models and 12 source corpora, distributing pivotal information lowers success by 22.9 percentage points versus giving it to one model, and forcing agents to extract it from source material lowers success by a further 19.9 points. Reinforcement learning on team decision loss improved final decisions and produced coordination even though the reward scored only the final decision.
- Tasks come from legal, finance, medical, and software corpora.
- Pivotal facts are necessary individually and sufficient together.
- A communication gap averages 22.9 percentage points.
- A retrieval gap averages a further 19.9 points.
- Two RL experiments improved decisions and induced coordination.
From the original article
1 min read at arxiv.orgMany real-world decisions require teams to integrate information distributed across participants. Yet AI agents are still primarily trained and benchmarked in isolation. We introduce TeamGym, an environment for evaluating and training language model-based agents on team decision-making problems constructed from real-world legal, finance, medical, and software engineering corpora.
By Chanwoo Park · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.