Library of Layers dynamically orders transformer layers at runtime
Original title: Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
LoL uses a single hyperparameter for its global layer library to generalize dense, recurrent, and mixture-of-experts architectures with distinct ordering for different tasks.
Existing language models execute layers in fixed order. Library of Layers (LoL) dynamically reorders transformer blocks during inference. It transitions between parameter reuse and sparsity and achieves competitive performance on text pretraining while showing adaptive behavior on synthetic and multilingual data.
- LoL has a single hyperparameter: the size of its global library of transformer blocks.
- It generalizes dense, depth-recurrent, and mixture-of-experts architectures.
- LoL applies distinct layer orderings for different solution strategies on synthetic data.
- Probes on router outputs distinguish safe from unsafe text with high AUROC.
From the original article
1 min read at huggingface.co
Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduce Library of Layers (LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand.
By Abdellah Rahmani, Sean McLeish, Ang Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.