Skip to content
NewA calmer, faster ZeroHour

PolyRoute benchmarks typed routing in 12 languages and code-mixed text

Original title: PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

Hugging Face daily papersby Mohammed Nayeem Khan, Sarun Kumar K S ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

A fine-tune scores 96.6% and TF-IDF with logistic regression 91.9%, while every text and label is LLM-generated.

PolyRoute is a released benchmark for a three-way department-routing decision: 322 native support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages, and 967 optional native training rows. Labels come from Qwen3.8-27B with a blind second pass that kept 91% of candidates; gemma4-31b agrees on 97.2% of test items. Across 13 systems the English-root checkpoint scores 61.2%, the shipped multilingual checkpoint 77.3%, and the best fine-tune 96.6%. On code-mixed text the English root’s 85% lead over the multilingual checkpoint’s 79% is not significant (p=0.134).

Unconfirmed: All texts and labels are LLM-generated, and the code-mixed gap is not statistically significant.

Key facts
  • Largest character 5-gram overlap between any test and training text is 0.33.
  • On four withheld languages the three multilingual systems score 77%, 90%, and 93%.
  • The split only resolves differences above about 4 points.
  • Code-mixed sets include Hinglish, Spanglish, and Arabizi.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

Typed-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick. We release PolyRoute: 322 native-language support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages (Hinglish, Spanglish, Arabizi), 967 optional native training rows, and a protocol for a three-way department-routing decision.

Hugging Face daily papers

By Mohammed Nayeem Khan, Sarun Kumar K S · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.