Planning to Learn
Horizon loss, a one-line cross-entropy modification, improves top-1 accuracy on ImageNet and MNIST, with gains increasing under label noise.
The paper reframes cross-entropy as 'patient accuracy' (total error an example pays as log-odds rise) versus exact policy gradient as the zero-horizon limit, exposing the myopia of exact gradients in classification. Truncating this total by remaining learning yields the horizon loss, which interpolates from cross-entropy toward exact policy gradient as training runs out. In a simple allocation model it provably escapes a trap that catches both endpoints. On MNIST and ImageNet with ResNet-50, ResNet-101, and ViT-S/16, horizon loss improves top-1 accuracy over cross-entropy at flat learning rates, with gains growing with label noise.
- Cross-entropy recast as patient accuracy; exact policy gradient is zero-horizon limit
- Horizon loss is a one-line change that interpolates between the two
- Improves top-1 accuracy on MNIST and ImageNet with ResNet-50/101 and ViT-S/16
- Accuracy gains grow as label noise increases
Full article209 words · extracted from arxiv.org · click to collapse
Policy-gradient methods are central to modern reinforcement learning, including LLM post-training. When they struggle, the usual suspects are exploration, credit assignment and action-sampling noise. Classification has none of them. A classifier is a policy whose expected reward, its \emph{expected accuracy}, is the probability it assigns to the correct label, and because that label is known, the policy gradient is exact and smooth. Yet exact policy gradient loses to cross-entropy, even on expected accuracy. The exact gradient is myopic: it values an update only by what it buys now, but each update also sets where the next one starts, so an update's value depends on how much learning remains. Viewed this way, cross-entropy is patient accuracy, the total error an example would pay if its log-odds rose at unit speed forever, while exact policy gradient is the zero-horizon limit. Truncating this total at the learning that remains yields the horizon loss, a one-line change that moves from cross-entropy toward exact policy gradient as training runs out. In a simple allocation model, it provably escapes the trap that catches each endpoint. On MNIST and on ImageNet with ResNet-50, ResNet-101 and ViT-S/16, the horizon loss improves top-1 accuracy over cross-entropy at a flat learning rate, and the gain grows with label noise.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.03667