The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The Functionalizer losslessly factors casing, diacritics and character repetition into reversible opcode prefixes, cutting tokenizer vocabulary needs by up to 19.7 percent.