The Surprising Effectiveness of Approximate Value Iteration in Self-Play
Minimal approximate value iteration self-play learns more accurate value functions than AlphaZero in Connect Four and Hex while cutting training and inference costs.
The paper trains a minimal self-play implementation of Approximate Value Iteration (AVI) without MCTS and uses ground-truth oracles for exact evaluation in Connect Four, 7x7 Hex, and synthetic games. AVI learns more accurate value functions than AlphaZero, and its one-step-lookahead greedy policies remain competitive with MCTS-based policies at substantially lower training and inference cost. Preliminary experiments on Othello and 9x9 Go show AVI trains stably on larger games, suggesting simpler approaches have become increasingly practical with modern deep-learning tools.
- AVI outperforms AlphaZero on value-function accuracy in moderate-sized games.
- One-step greedy policies stay competitive with MCTS at much lower cost.
- Stable training shown on Othello and 9x9 Go in preliminary tests.
- AI models
- AlphaZero
From the original article
1 min read at arxiv.orgCombining search with function approximation has driven major advances in game-playing programs, making self-play algorithms more competitive than ever. Still, the computational overhead of the most popular methods, based on Monte Carlo Tree Search (MCTS), can be substantial.
By Raphael Boige · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.