Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy
Researchers added Greek to the Cosmos3 vision-language-action policy using only machine-rephrased instructions, finding bilingual training reaches roughly two fifths of English performance.
The paper studies localizing the open Cosmos3 vision-language-action robot policy to Greek without architectural changes, using machine-rephrased instructions only. Bilingual training yields a consistent 6.7-7.1 point margin over controls on a 90-task, three-seed evaluation suite, while Greek-only training gains at most 2.7 points. Several common evaluation instruments, including color-histogram metrics and single-goal benchmarks, produced false conclusions, and results were dominated by seed variation. The authors recommend building guaranteed-null baselines and replicating low-resource-language results across seeds.
- Bilingual training improves Greek robot policy performance by 6.7-7.1 points over controls.
- Multilingual text tower without Greek data stays at wrong-instruction floor performance.
- Single-run comparisons unreliable due to seed variation; three seeds per arm needed.
- Training on seven phrasings per task approximately halves translator-phrasing overfitting penalty.
Full article194 words · extracted from huggingface.co · click to collapse
Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist for most languages. We study the addition of Greek to an open vision-language-action stack using only machine-rephrased instructions and no architecture changes. The main challenge is measurement rather than translation. Several plausible instruments produce false conclusions: a color-histogram metric rewards noise, a single-goal benchmark scores 84.6% under correct Greek and 82.6% under deliberately wrong instructions, training loss fails to predict Greek success, and single-run comparisons are dominated by seed variation. On a discriminative ninety-task suite with three seeds per arm, a multilingual text tower without Greek demonstrations remains at its wrong-instruction floor, while Greek-only training exceeds its control by at most 2.7 points. Bilingual training yields a consistent 6.7-7.1 point margin over its control and reaches about two fifths of English performance. The policy also overfits the translator's phrasing; training on seven phrasings per task approximately halves this penalty. Warm-starting from a language-adapted world model and unfreezing the text tower both degrade performance. The results support two practical requirements for low-resource robot-policy localization: build a guaranteed null before trusting a metric, and replicate low-resource-language results across seeds.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.07470