Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
Fixed 16-bit token codes let 1.7B decoder-only models learn substantial language ability without trainable input embeddings.
1 stories, newest first · one card per story
Fixed 16-bit token codes let 1.7B decoder-only models learn substantial language ability without trainable input embeddings.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views