Multilingual GSM-Symbolic: What determines capability transfer across languages?
Multilingual GSM-Symbolic finds model size, resources, reasoning, and typology drive cross-lingual math transfer.
The paper introduces Multilingual GSM-Symbolic, an extensible math benchmark with 30,000 item-matched question-answer pairs across 15 languages, using symbolic templates that can generate millions of variations. It estimates the largest drivers of cross-lingual transfer as model size (β=1.77), language resource level (β=0.77), reasoning (β=0.67), and typological distance (β=-0.25). A 32B model in Marathi performs like a 10B model in English. The framework explains 92% of between-language variation and predicts unseen-language performance within 6.0 percentage points (r=.96).
- Benchmark covers 15 languages and 30,000 matched math pairs.
- Model size is the strongest predictor, with beta 1.77.
- A 32B model in Marathi resembles a 10B model in English.
- Ten target-language templates reduce error to 4.19 percentage points.
From the original article
1 min read at huggingface.co
We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datasets and rarely examine its determinants jointly. Identifying what predicts transfer would let us avoid exhaustive evaluation across all language pairs and let developers target the factors that limit performance in low-resource languages.
By Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.