Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
Fixed 16-bit token codes let 1.7B decoder-only models learn substantial language ability without trainable input embeddings.
Researchers trained three decoder-only language models from scratch with the same tokenizer, backbone, untied output head, and recipe, targeting 100 billion prediction tokens. Input interfaces were a learned embedding table, canonical 16-bit token-ID codes, and one fixed invertible recoding over GF(2), repeated to model width with no extra trainable input projection. Fixed-code models reached 52.40% HellaSwag, 70.51% PIQA, and 42.75% LAMBADA, while the learned-input control scored higher on several tasks. Removing the table cut 100.7 million parameters, yielding 1.711B-parameter models and showing independently trainable token vectors are not required for those capabilities.
- Three 1.7B-class models trained from scratch on about 100B tokens.
- Fixed 16-bit codes and a GF(2) recoding replace learned input embeddings.
- Canonical codes score 52.40% HellaSwag, 70.51% PIQA, and 42.75% LAMBADA.
- Learned embeddings still outperform; findings show viability, not parity.
- Fixed interfaces remove 100.7 million trainable parameters.
From the original article
1 min read at huggingface.co
A trainable input embedding table assigns each vocabulary item an independently adjustable vector. We investigate whether this token-specific parameterization is required for substantial language-modeling capability, or whether a shared Transformer can learn from fixed token identities.
By A. Bochkov · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.