Distance generalization in transformers: why bother with positional encoding?
arXiv study uses synthetic delay-copy tasks to show how RoPE, ALiBi, NoPE and training data diversity affect transformers' distance generalization.
The paper studies distance generalization in transformers: extrapolating when inter-token distances change between training and inference while context length stays fixed. Using two synthetic delay-copy tasks with finite source-recall distances, the authors test models on unseen delays. They investigate whether positional encodings such as RoPE and ALiBi outperform no positional encoding (NoPE), how the diversity of training distances affects performance, and when distance transfer learning is positive or negative.
- Studies distance generalization: fixed context length but changed inter-token distances between training and testing.
- Compares RoPE, ALiBi, and NoPE positional encodings on fully and selectively copied delay tasks.
- Tests how the number of training distances and transfer direction affect extrapolation.
- Findings emphasize understanding underlying mechanisms over relying on the encoding choice.
From the original article
1 min read at arxiv.orgOut-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length.
By Daniel Henrik Nevermann · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.