PolyRoute benchmarks typed routing in 12 languages and code-mixed text
Original title: PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard
A fine-tune scores 96.6% and TF-IDF with logistic regression 91.9%, while every text and label is LLM-generated.
PolyRoute is a released benchmark for a three-way department-routing decision: 322 native support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages, and 967 optional native training rows. Labels come from Qwen3.8-27B with a blind second pass that kept 91% of candidates; gemma4-31b agrees on 97.2% of test items. Across 13 systems the English-root checkpoint scores 61.2%, the shipped multilingual checkpoint 77.3%, and the best fine-tune 96.6%. On code-mixed text the English root’s 85% lead over the multilingual checkpoint’s 79% is not significant (p=0.134).
Unconfirmed: All texts and labels are LLM-generated, and the code-mixed gap is not statistically significant.
- Largest character 5-gram overlap between any test and training text is 0.33.
- On four withheld languages the three multilingual systems score 77%, 90%, and 93%.
- The split only resolves differences above about 4 points.
- Code-mixed sets include Hinglish, Spanglish, and Arabizi.
From the original article
1 min read at huggingface.coTyped-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick. We release PolyRoute: 322 native-language support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages (Hinglish, Spanglish, Arabizi), 967 optional native training rows, and a protocol for a three-way department-routing decision.
By Mohammed Nayeem Khan, Sarun Kumar K S · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.