Text encoding changes byte-level language model accuracy by language
Original title: When does text encoding matter for language modeling?
Aether uses script-specific encodings with UTF-8 fallback; Chinese and Korean gain, while Georgian and Tamil do not.
The paper studies how text encoding affects byte-level language models, comparing UTF-8 with Unicode-based, language-specific, and custom encodings. The authors propose Aether, which uses script-specific encodings for a target setting and falls back to UTF-8 for full Unicode coverage, and model four orthographically diverse languages. They report large gains from compressive encodings for Chinese and Korean, and little sensitivity for Georgian and Tamil, and argue encoding should be chosen per language.
Unconfirmed: Performance differences are reported by the authors and not independently confirmed.
- Aether keeps Unicode coverage through UTF-8 fallbacks.
- Information-dense scripts benefit from compressive encodings, the authors report.
- Georgian and Tamil are largely insensitive to encoding choice.
- The work compares Unicode, language-specific, and custom encodings.
From the original article
1 min read at arxiv.orgUTF-8's role as the de facto encoding standard in language models introduces cross-lingual inequities due to its variable-length design. However, the effect that this encoding choice has on model performance remains largely unexamined.
By Eugene Jang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.