PAR aligns looped model steps for longer state tracking
On swap-S5, a Qwen2.5-0.5B model trained on at most 32 moves reaches 95.5% full-state accuracy at 512 moves.
Stories per day · 30 days
2 RSS
2 stories, newest first · one card per story
Stories tagged #length-generalization
Distance generalization in transformers: why bother with positional encoding?
arXiv study uses synthetic delay-copy tasks to show how RoPE, ALiBi, NoPE and training data diversity affect transformers' distance generalization.