AdaBPE adapts BPE merge thresholds for mixed language and LaTeX
Original title: Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora
On five LaTeX and natural-language datasets, the method improves encoding and domain-specific model performance without hurting general language.
Researchers introduce Adaptive Frequency Threshold Byte Pair Encoding (AdaBPE), a tokenizer that sets minimum frequency thresholds by candidate token length and by whether a pattern is general language or formal notation such as LaTeX. The method is meant to give domain-specific symbols vocabulary capacity even when they are a small share of training data. Evaluation across five LaTeX and natural-language datasets reports consistent gains in encoding efficiency and downstream language-model performance on domain content without compromising general-language performance.
- AdaBPE varies frequency thresholds by token length and pattern type.
- LaTeX is the formal-language case used to distinguish domain notation.
- Five datasets cover LaTeX and natural language.
- Gains are reported for encoding efficiency and domain-specific model performance.
From the original article
1 min read at arxiv.orgModern language models rely on subwording tokenization to define the discrete units of input processing, typically using methods like Byte Pair Encoding (BPE) that treat whitespace as a fixed semantic boundary. This design is inherently limited, as many meaningful concepts---including multiword expressions and the structured environments of formal languages like LaTeX---frequently span across whitespaces.
By Pietro Di Nicco · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.