ParaTTT parallelizes deep nonlinear test-time training memories
Original title: ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
Two refinement steps at 4K context evaluate the inner-loop trajectory 32× faster than TTT-MLP at matched error.
A paper introduces ParaTTT, an algorithm that parallelizes arbitrarily deep nonlinear memories used in test-time training layers. It iteratively refines estimates of the inner-loop gradient trajectory with three chunkwise-parallel passes, and the authors prove geometric convergence for a two-layer variant under contraction conditions. Models at 340M and 1.3B parameters match or beat Gated DeltaNet and Mamba-2 and stay competitive with same-size Transformers, with better recall than linear recurrences.
- Decoding needs no refinement iterations; two iterations suffice in practice.
- The full 340M model trains 4.9× faster than the compared TTT-MLP setup.
- The exact online gradient-descent recurrence is a fixed point of the iteration.
From the original article
1 min read at arxiv.orgTest-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent.
By Noel Loo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.