All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
ScriptMoE recognizes scene text across 229 languages, beating per-script models and a vision-language baseline.
Researchers introduce ScriptMoE, an all-in-one multilingual scene-text recognizer that shares one visual encoder and uses a sparse, script-aware mixture-of-experts decoder. Training uses TextMuSS-10M, a synthetic dataset spanning 10 scripts and 229 languages. On TextMuSS-Bench (10,899 images) it scores 82.06%, 1.31 points above the strongest scene-text baseline. Replacing the recognizer in PP-OCRv5 lifts CC-OCR end-to-end F1 from 65.71% to 80.89%, slightly above the best vision-language model at far fewer parameters.
- TextMuSS-10M covers 10 scripts and 229 languages with synthetic scene text.
- ScriptMoE routes each image to the top-2 script-aligned experts plus a shared expert.
- Accuracy reaches 82.06% on TextMuSS-Bench, 1.31 points above the best STR baseline.
- Swapping ScriptMoE into PP-OCRv5 raises CC-OCR F1 from 65.71% to 80.89%, slightly above the best VLM.
Full article214 words · extracted from huggingface.co · click to collapse
Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.24058