Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
Small autoregressive transformers recover unseen chaotic dynamics, including a near-Feigenbaum logistic-map scaling ratio.
Researchers train small autoregressive transformers from scratch on trajectories from restricted parameter regimes of nonlinear systems, including logistic and sine maps, the Lorenz system, and the generalized Hopf system. Under closed-loop evaluation far outside the training distribution, the models recover period-doubling cascades, chaotic dynamics, and attractor structure. On the logistic map, a transformer reproduces period doublings up to period 128 and a finite-order scaling ratio of 4.6687, within 5×10⁻⁴ of the Feigenbaum constant. Causal interventions show how control-parameter information flows through attention into state prediction.
- Small transformers trained from scratch on partial dynamical trajectories.
- Systems include logistic and sine maps, Lorenz, and generalized Hopf.
- Out-of-distribution closed-loop runs recover cascades, chaos, and attractors.
- Logistic-map scaling ratio 4.6687 matches Feigenbaum within 0.0005.
- Causal interventions trace how attention uses control parameters.
From the original article
1 min read at huggingface.co
Autoregressive models are trained to predict a system's behavior one step at a time, and recursive generation allows the learned dynamics to unfold over long horizons. To what extent can such dynamics learned from local observations recover broader organization of an underlying system that was only partially observed during training?
By Yilun Liu, Yi Zhang, Ganyu Wu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.