Sindhi SentencePiece Unigram tokenizer beats 14 baselines on intrinsic metrics
Original title: A Corpus-Driven SentencePiece Unigram Tokenizer for Sindhi: Design and Benchmark Against Fourteen Existing Tokenizers
At 64k vocabulary it averages 1.147 tokens per word and keeps 88.6% of words as single tokens, with no round-trip character loss.
Researchers trained SentencePiece Unigram tokenizers with byte fallback at four vocabulary sizes, from 32k to 150k, on about 1.10 million Sindhi sentences plus a smaller repeated Urdu, Persian and Arabic slice. They compared them with fourteen existing tokenizers on FLORES+, Sindhi Wikipedia, a news sentiment corpus and a contemporary news corpus using fertility, single-token retention, UNK rate, byte-fallback rate and round-trip character loss. At 64k and above the new tokenizer is best or tied on fertility, single-token retention and UNK rate; downstream effects were not tested. Code and tokenizer files are public.
Unconfirmed: No significance tests are reported, and effects on downstream tasks remain untested.
- Vocabulary sizes span 32k to 150k, trained on four public sources.
- At 64k, XLM-RoBERTa averages 1.512 tokens per word versus 1.147 for the new tokenizer.
- The margin over the best Sindhi-adapted baseline is 0.002 fertility on FLORES+ at 64k.
- Two existing tokenizers silently delete up to 0.85% of characters; seven others lose under 0.1%.
- At 32k the new tokenizer trails one baseline on FLORES+.
From the original article
1 min read at arxiv.orgSindhi is a low-resource Indo-Aryan language written mainly in an extended Perso-Arabic script. Several Sindhi tokenizers have been released on public model hubs, but they have not been compared with one another on shared data. We train SentencePiece Unigram tokenizers with byte fallback at four vocabulary sizes (32k to 150k) on about 1.
By Ali Nawaz · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.