The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The Functionalizer losslessly factors casing, diacritics and character repetition into reversible opcode prefixes, cutting tokenizer vocabulary needs by up to 19.7 percent.
The Functionalizer is a lossless pre-tokenizer that decomposes orthographic variations into a canonical base token prefixed by reversible transformation operators encoded in the Unicode Private Use Area, covering casing, 13 diacritic operations, and repetition. It reduces vocabulary slot requirements by up to 19.7 percent across natural language and code corpora. On 98M-parameter GPT-2 models, it improves Python code syntax validity from 7.70 to 9.12 percent while reducing duplicate n-gram repetition in prose.
- Reversible opcode/operand prefixes replace separate vocabulary entries per orthographic variant
- Vocabulary slot requirements cut by up to 19.7 percent
- 98M-parameter GPT-2 experiments show better Python syntax validity and less repetition
Full article173 words · extracted from huggingface.co · click to collapse
Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization. We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area. We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible. Across natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained exhaustion conditions, reducing actual vocabulary slot requirements by up to 19.7%. Downstream evaluations on 98M-parameter GPT-2 models show that the Functionalizer improves Python code syntax validity (9.12% vs. 7.70%) while reducing duplicate n-gram repetition in natural language prose. These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.15991