PAR aligns looped model steps for longer state tracking
On swap-S5, a Qwen2.5-0.5B model trained on at most 32 moves reaches 95.5% full-state accuracy at 512 moves.
Stories per day · 30 days
4 stories, newest first · one card per story
Stories tagged #state-tracking
Recurrent Looped Transformer
Recurrent Looped Transformers track algorithmic state far beyond training length via per-token feedback.
Pretraining Latent Information Feedback Transformers with Teacher Supervision
LIFT pretrains Transformers to recycle latent state and beats token-matched baselines from 135M to 1B.
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Complex KDA extends Kimi Delta Attention so one transition can represent 2D rotations and stronger state tracking.