Distance generalization in transformers: why bother with positional encoding?
arXiv study uses synthetic delay-copy tasks to show how RoPE, ALiBi, NoPE and training data diversity affect transformers' distance generalization.
The paper studies distance generalization in transformers: extrapolating when inter-token distances change between training and inference while context length stays fixed. Using two synthetic delay-copy tasks with finite source-recall distances, the authors test models on unseen delays. They investigate whether positional encodings such as RoPE and ALiBi outperform no positional encoding (NoPE), how the diversity of training distances affects performance, and when distance transfer learning is positive or negative.
- Studies distance generalization: fixed context length but changed inter-token distances between training and testing.
- Compares RoPE, ALiBi, and NoPE positional encodings on fully and selectively copied delay tasks.
- Tests how the number of training distances and transfer direction affect extrapolation.
- Findings emphasize understanding underlying mechanisms over relying on the encoding choice.
Full article139 words · extracted from arxiv.org · click to collapse
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.11913