Choosing LLMs for Strategic Agent Interactions
Selecting LLMs for strategic games is game- and opponent-specific; a stationary bandit beats more complex selectors but not the best fixed model.
The paper treats choosing an LLM for strategic play as a meta-game layered on a base game, where a selector fields a model from a portfolio against a latent, online-learned payoff matrix. Across five games and ten LLMs, strategic competence varies strongly with the game setup and the opponent, and reliable evaluation needs far more seed runs than typical benchmarks. Selectors based on query embeddings or LLM self-reflection do not consistently recover the strategic signal. A lightweight multi-armed bandit is more reliable across games but still falls short of the best fixed model, and non-stationary or adversarial bandit variants do not beat a stationary baseline.
- Strategic selection is framed as a meta-game over a latent payoff matrix learned online.
- Evaluation across five games and ten LLMs shows competence depends on game and opponent.
- Embedding and self-reflection selectors do not reliably capture the strategic signal.
- A stationary multi-armed bandit is more reliable but still trails the best fixed model.
From the original article
1 min read at huggingface.co
Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure.
By Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.