onóna uses twin tied embeddings to cut validation loss
Original title: onóna: Twin Embeddings with Tied Unembeddings
At matched token budgets it lowered loss in 11 of 12 runs and raised average accuracy in 10 of 12.
The paper introduces onóna, a two-table parameterization that keeps input-output weight sharing while mixing two softmax distributions. A gate conditioned only on the current input token sets the mixture and can be precomputed. Evaluations cover Dense, Gated DeltaNet hybrid, mixture-of-experts and Attention Residual models at 142M, 330M and 1B. The extra cost is one embedding table and one vocabulary projection.
- The Transformer body is unchanged.
- The 1B MoE model is the loss exception.
- Average accuracy is over eight downstream tasks.
- Input is the average of two embedding tables.
From the original article
1 min read at huggingface.co
Weight tying reduces the parameter cost of language models, but the usual tied output head is limited to a single softmax. We introduce onóna, a two-table parameterization that increases output expressiveness while preserving input-output weight sharing. The Transformer takes the average of two embedding tables as input, and the output head combines the softmax distributions defined by those same tables.
By Cheng Luo, Xinnan Dai, Zhongzhu Zhou · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.