The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The Functionalizer losslessly factors casing, diacritics and character repetition into reversible opcode prefixes, cutting tokenizer vocabulary needs by up to 19.7 percent.
The Functionalizer is a lossless pre-tokenizer that decomposes orthographic variations into a canonical base token prefixed by reversible transformation operators encoded in the Unicode Private Use Area, covering casing, 13 diacritic operations, and repetition. It reduces vocabulary slot requirements by up to 19.7 percent across natural language and code corpora. On 98M-parameter GPT-2 models, it improves Python code syntax validity from 7.70 to 9.12 percent while reducing duplicate n-gram repetition in prose.
- Reversible opcode/operand prefixes replace separate vocabulary entries per orthographic variant
- Vocabulary slot requirements cut by up to 19.7 percent
- 98M-parameter GPT-2 experiments show better Python syntax validity and less repetition
From the original article
1 min read at huggingface.co
Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization.
By Connor Makowski, Willem Guter · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.