Skip to content
NewA calmer, faster ZeroHour

Sindhi SentencePiece Unigram tokenizer beats 14 baselines on intrinsic metrics

Original title: A Corpus-Driven SentencePiece Unigram Tokenizer for Sindhi: Design and Benchmark Against Fourteen Existing Tokenizers

arXiv cs.AI / cs.LG / cs.CLby Ali Nawaz ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

At 64k vocabulary it averages 1.147 tokens per word and keeps 88.6% of words as single tokens, with no round-trip character loss.

Researchers trained SentencePiece Unigram tokenizers with byte fallback at four vocabulary sizes, from 32k to 150k, on about 1.10 million Sindhi sentences plus a smaller repeated Urdu, Persian and Arabic slice. They compared them with fourteen existing tokenizers on FLORES+, Sindhi Wikipedia, a news sentiment corpus and a contemporary news corpus using fertility, single-token retention, UNK rate, byte-fallback rate and round-trip character loss. At 64k and above the new tokenizer is best or tied on fertility, single-token retention and UNK rate; downstream effects were not tested. Code and tokenizer files are public.

Unconfirmed: No significance tests are reported, and effects on downstream tasks remain untested.

Key facts
  • Vocabulary sizes span 32k to 150k, trained on four public sources.
  • At 64k, XLM-RoBERTa averages 1.512 tokens per word versus 1.147 for the new tokenizer.
  • The margin over the best Sindhi-adapted baseline is 0.002 fertility on FLORES+ at 64k.
  • Two existing tokenizers silently delete up to 0.85% of characters; seven others lose under 0.1%.
  • At 32k the new tokenizer trails one baseline on FLORES+.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Sindhi is a low-resource Indo-Aryan language written mainly in an extended Perso-Arabic script. Several Sindhi tokenizers have been released on public model hubs, but they have not been compared with one another on shared data. We train SentencePiece Unigram tokenizers with byte fallback at four vocabulary sizes (32k to 150k) on about 1.

arXiv cs.AI / cs.LG / cs.CL

By Ali Nawaz · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.