Nestor nested sparsity beats MoE transformers at equal compute
Original title: Nested Sparsity for Scalable Pretraining and Continual Learning
Attention-side sparsity nests with mixture-of-experts sparsity and adapts to new domains with less forgetting.
Nestor adds conditional computation that generalizes the attention operator and nests with mixture-of-experts sparsity. In language-model pretraining it outperforms a conventional MoE transformer at equal compute and equal sparsity. It also adapts to longer contexts more effectively and to new domains with less forgetting on old domains. The authors say the gains persist as model size, dataset size, and context length grow. Code and checkpoints are on GitHub.
- Sparsity comes from an attention-side mechanism, not only the MoE operator.
- Continual pretraining is a stated strength versus conventional MoEs.
- Checkpoints are published at github.com/Ma-Lab-Berkeley/Nestor.
- Products
- Nestor
- Organizations
- Ma Lab Berkeley
From the original article
1 min read at huggingface.co
Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model weights is used in each forward pass. In MoE transformers, this sparsity arises in the MoE operator, which generalizes the multi-layer perceptron in the classical transformer.
By Tianzhe Chu, Druv Pai, Yi Ma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.